Stanford University CS224d: Deep Learning for Natural Language Processing

http://cs224d.stanford.edu/

Through lectures and programming assignments students will learn the necessary engineering tricks for making neural networks work on practical problems.

http://cs224d.stanford.edu/syllabus.html

共変量シフト下での教師付き学習

https://www.jstage.jst.go.jp/article/jnns/13/3/13_3_111/_pdf

本稿では，共変量シフト下での教師付き学習の最近の研究成果を概説する．

共変量シフト適応 - 東京大学 杉山・佐藤研究室

http://www.ms.k.u-tokyo.ac.jp/research-jp.html#theory-different-distributions-covariate-shift

入出力関係（関数）は訓練時とテスト時で変わらないが， 訓練入力とテスト入力は異なる確率分布に従うという状況を共変量シフトとよぶ．

共変量シフトの簡単な解説（図付き）。

[1502.03167] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

http://arxiv.org/abs/1502.03167

Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch.

C++11のスレッド、アフィニティ、ハイパースレッディング | プログラミング | POSTD

http://postd.cc/c11-threads-affinity-and-hyperthreading/

C++のスレッディングのライブラリとPOSIX呼び出しによって、所定のマシンにおけるCPUの使い方を微調整し、どのCPUでどのスレッドを実行するかまで正確に指定します。

[関連]

• std::thread::hardware_concurrency - cppreference.com
  http://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency
• Processor affinity - Wikipedia, the free encyclopedia
  https://en.wikipedia.org/wiki/Processor_affinity
• Portable Hardware Locality (hwloc)
  https://www.open-mpi.org/projects/hwloc/

Automatic Colorization

http://tinyclouds.org/colorize/

This project is an attempt to use modern deep learning techniques to automatically colorize black and white photos.

http://tinyclouds.org/

Ryan Dahl

Japanese Cheat Sheet Pack by Nihonshock.com » Basic Japanese

http://cheatsheets.nihonshock.com/sheets/basic-japanese/

Nihonshock.com has offered and will continue to offer the digital version of its Basic Japanese cheat sheet as a free resource for Japanese learners.

google/brotli · GitHub

https://github.com/google/brotli/

Brotli is a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling, with a compression ratio comparable to the best currently available general-purpose compression methods. It is similar in speed with deflate but offers more dense compression.

GitHub - google/zopfli: Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

https://github.com/google/zopfli

Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Google Open Source Blog: Introducing Brotli: a new compression algorithm for the internet

http://google-opensource.blogspot.jp/2015/09/introducing-brotli-new-compression.html

While Zopfli is Deflate-compatible, Brotli is a whole new data format. This new format allows us to get 20–26% higher compression ratios over Zopfli.

Go - Luceneで使われてるFSTを実装してみた（正規表現マッチ：VMアプローチへの招待） - Qiita

http://qiita.com/ikawaha/items/be95304a803020e1b2d1

FST というのは Finite State Transducer (有限状態変換機) の略で，有限オートマトンに出力がついたものです．変換機とか言うと小難しい感じがするかもしれませんが，文字列を入力として，文字列を返す単純な機構です．（機能としては map[string]string みたいなものです)．

udp/json-parser · GitHub

https://github.com/udp/json-parser

Very low footprint JSON parser written in portable ANSI C.

Runtime Options
settings |= json_enable_comments;
Enables C-style // line and /* block */ comments.

コメント付きのJSONを扱うことができる、C/C++言語用のJSONパーサのソースコード。

cedar - C++ implementation of efficiently-updatable double-array trie

http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/cedar/

Cedar implements an updatable double-array trie [1,2,3], which offers fast update/lookup for skewed queries in real-world data, e.g., counting words in text or mining conjunctive features in a classifier.

Quantum Annealing: Quantum computing for combinatorial optimization problems

http://www.stat.phys.titech.ac.jp/~nishimori/QA/q-annealing_e.html

量子アニーリング（西森秀稔）
http://www.stat.phys.titech.ac.jp/~nishimori/QA/q-annealing.html

量子アニーリング(Quantum annealing)は，量子効果を制御して，多変数1価関数（目的関数）の最小値を探す問題（最適化問題）を解く手法である。 パターン認識，自然言語処理，医療診断，金融商品開発その他への応用を持つ機械学習を始めとする多くの重要な課題が最適化問題として定式化できるため，最適化問題の効率的な解法は社会的に大きなインパクトを持つ。

robertsdionne/neural-network-papers · GitHub

https://github.com/robertsdionne/neural-network-papers

neural-network-papers

Table of Contents
    Other Lists
    Surveys
    Books
    Datasets
    Pretrained Models
    Programming Frameworks
    Learning to Compute
    Natural Language Processing
    Convolutional Neural Networks
    Recurrent Neural Networks
    Convolutional Recurrent Neural Networks
    Adversarial Neural Networks
    Autoencoders
    Restricted Boltzmann Machines
    Biologically Plausible Learning
    Supervised Learning
    Unsupervised Learning
    Reinforcement Learning
    Theory
    Quantum Computing
    Training Innovations
    Parallel Training
    Weight Compression
    Numerical Precision
    Numerical Optimization
    Motion Planning
    Simulation
    Hardware
    Cognitive Architectures
    Computational Creativity
    Cryptography
    Distributed Computing
    Clustering

Neural network paper list | Eniod's Blog

https://haduonght.wordpress.com/2015/12/23/neural-network-paper-list/

Based on https://github.com/robertsdionne/neural-network-papers

Computational Network Toolkit (CNTK) - Home

https://cntk.codeplex.com/

CNTK, the Computational Network Toolkit by Microsoft Research, is a unified deep-learning toolkit

CNTK allows to easily realize and combine popular model types such as feed-forward DNNs, convolutional nets (CNNs), and recurrent networks (RNNs/LSTMs). It implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers.

Microsoft Research による、Windows と Linux に対応した deep learning のツールキット。
下記ページに速度性能を比較した記事がある。

Microsoft Computational Network Toolkit offers most efficient distributed deep learning computational performance
http://blogs.technet.com/b/inside_microsoft_research/archive/2015/12/07/microsoft-computational-network-toolkit-offers-most-efficient-distributed-deep-learning-computational-performance.aspx

[1506.03340] Teaching Machines to Read and Comprehend

http://arxiv.org/abs/1506.03340

In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data.

The Impatient Reader
The Attentive Reader is able to focus on the passages of a context document that are most likely to inform the answer to the query. We can go further by equipping the model with the ability to reread from the document as each query token is read.

文書を学習して、穴埋め問題を解く。Impatient Reader と Attentive Reader の比較。

スライドはこの記事を参照。

NTCIR-12 Task on Short Text Conversation

http://ntcir12.noahlab.com.hk/japanese/stc-jpn.htm

Dataset of Japanese task

The input data will be those randomly sampled from tweets in the year 2015. The pool of tweets (the target for extraction) is the randomly sampled tweet pairs (mention-reply pairs) in the year 2014. The size of the pool is just over one million; that is 500K pairs.

The following data will be provided from the organizers:
(1) Twitter data (by using their IDs) 1M in size
(2) Development data. Input samples and output samples annotated with reference labels. Here, the number of annotators is ten.

対話ツイートのツイートIDのみのペア。ツイート本文はなし。

Teaching Machines to Read and Comprehend (slide)

http://lxmls.it.pt/2015/lxmls15.pdf

Conclusion

Summary
* supervised machine reading is a viable research direction with the available data,
* LSTM based recurrent networks constantly surprise with their ability to encode dependencies in sequences,
* attention is a very effective and exible modelling technique.

Future directions
* more and better data, corpus querying, and cross document queries,
* recurrent networks incorporating long term and working memory are well suited to NLU task.

Lisbon Machine Learning School 2015 のスライド。トピックは自然言語処理。