google/brotli · GitHub

https://github.com/google/brotli/

Brotli is a generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling, with a compression ratio comparable to the best currently available general-purpose compression methods. It is similar in speed with deflate but offers more dense compression.

GitHub - google/zopfli: Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

https://github.com/google/zopfli

Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Google Open Source Blog: Introducing Brotli: a new compression algorithm for the internet

http://google-opensource.blogspot.jp/2015/09/introducing-brotli-new-compression.html

While Zopfli is Deflate-compatible, Brotli is a whole new data format. This new format allows us to get 20–26% higher compression ratios over Zopfli.

Go - Luceneで使われてるFSTを実装してみた（正規表現マッチ：VMアプローチへの招待） - Qiita

http://qiita.com/ikawaha/items/be95304a803020e1b2d1

FST というのは Finite State Transducer (有限状態変換機) の略で，有限オートマトンに出力がついたものです．変換機とか言うと小難しい感じがするかもしれませんが，文字列を入力として，文字列を返す単純な機構です．（機能としては map[string]string みたいなものです)．

udp/json-parser · GitHub

https://github.com/udp/json-parser

Very low footprint JSON parser written in portable ANSI C.

Runtime Options
settings |= json_enable_comments;
Enables C-style // line and /* block */ comments.

コメント付きのJSONを扱うことができる、C/C++言語用のJSONパーサのソースコード。

cedar - C++ implementation of efficiently-updatable double-array trie

http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/cedar/

Cedar implements an updatable double-array trie [1,2,3], which offers fast update/lookup for skewed queries in real-world data, e.g., counting words in text or mining conjunctive features in a classifier.

Quantum Annealing: Quantum computing for combinatorial optimization problems

http://www.stat.phys.titech.ac.jp/~nishimori/QA/q-annealing_e.html

量子アニーリング（西森秀稔）
http://www.stat.phys.titech.ac.jp/~nishimori/QA/q-annealing.html

量子アニーリング(Quantum annealing)は，量子効果を制御して，多変数1価関数（目的関数）の最小値を探す問題（最適化問題）を解く手法である。 パターン認識，自然言語処理，医療診断，金融商品開発その他への応用を持つ機械学習を始めとする多くの重要な課題が最適化問題として定式化できるため，最適化問題の効率的な解法は社会的に大きなインパクトを持つ。

robertsdionne/neural-network-papers · GitHub

https://github.com/robertsdionne/neural-network-papers

neural-network-papers

Table of Contents
    Other Lists
    Surveys
    Books
    Datasets
    Pretrained Models
    Programming Frameworks
    Learning to Compute
    Natural Language Processing
    Convolutional Neural Networks
    Recurrent Neural Networks
    Convolutional Recurrent Neural Networks
    Adversarial Neural Networks
    Autoencoders
    Restricted Boltzmann Machines
    Biologically Plausible Learning
    Supervised Learning
    Unsupervised Learning
    Reinforcement Learning
    Theory
    Quantum Computing
    Training Innovations
    Parallel Training
    Weight Compression
    Numerical Precision
    Numerical Optimization
    Motion Planning
    Simulation
    Hardware
    Cognitive Architectures
    Computational Creativity
    Cryptography
    Distributed Computing
    Clustering

Neural network paper list | Eniod's Blog

https://haduonght.wordpress.com/2015/12/23/neural-network-paper-list/

Based on https://github.com/robertsdionne/neural-network-papers

Computational Network Toolkit (CNTK) - Home

https://cntk.codeplex.com/

CNTK, the Computational Network Toolkit by Microsoft Research, is a unified deep-learning toolkit

CNTK allows to easily realize and combine popular model types such as feed-forward DNNs, convolutional nets (CNNs), and recurrent networks (RNNs/LSTMs). It implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers.

Microsoft Research による、Windows と Linux に対応した deep learning のツールキット。
下記ページに速度性能を比較した記事がある。

Microsoft Computational Network Toolkit offers most efficient distributed deep learning computational performance
http://blogs.technet.com/b/inside_microsoft_research/archive/2015/12/07/microsoft-computational-network-toolkit-offers-most-efficient-distributed-deep-learning-computational-performance.aspx

[1506.03340] Teaching Machines to Read and Comprehend

http://arxiv.org/abs/1506.03340

In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data.

The Impatient Reader
The Attentive Reader is able to focus on the passages of a context document that are most likely to inform the answer to the query. We can go further by equipping the model with the ability to reread from the document as each query token is read.

文書を学習して、穴埋め問題を解く。Impatient Reader と Attentive Reader の比較。

スライドはこの記事を参照。

NTCIR-12 Task on Short Text Conversation

http://ntcir12.noahlab.com.hk/japanese/stc-jpn.htm

Dataset of Japanese task

The input data will be those randomly sampled from tweets in the year 2015. The pool of tweets (the target for extraction) is the randomly sampled tweet pairs (mention-reply pairs) in the year 2014. The size of the pool is just over one million; that is 500K pairs.

The following data will be provided from the organizers:
(1) Twitter data (by using their IDs) 1M in size
(2) Development data. Input samples and output samples annotated with reference labels. Here, the number of annotators is ten.

対話ツイートのツイートIDのみのペア。ツイート本文はなし。

Teaching Machines to Read and Comprehend (slide)

http://lxmls.it.pt/2015/lxmls15.pdf

Conclusion

Summary
* supervised machine reading is a viable research direction with the available data,
* LSTM based recurrent networks constantly surprise with their ability to encode dependencies in sequences,
* attention is a very effective and exible modelling technique.

Future directions
* more and better data, corpus querying, and cross document queries,
* recurrent networks incorporating long term and working memory are well suited to NLU task.

Lisbon Machine Learning School 2015 のスライド。トピックは自然言語処理。

Computational Linguistics and Deep Learning

http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00239

Christopher D. Manning
Stanford University
MIT Press Journals - Computational Linguistics - Early Access - Citation

Deep Learningの教材 - こんな夢を見た

http://hytae.hatenablog.com/entry/2015/11/14/Deep_Learning%E3%81%AE%E6%95%99%E6%9D%90

Deep Learningを勉強するにあたって内容がまとまっている教材についてリスト化しました。

Marvin: Deep Learning in N Dimensions

http://marvin.is/

Marvin was born to be hacked, relying on few dependencies and basic C++. All code lives in two files (marvin.hpp and marvin.cu) and all numbers take up two bytes (FP16).

Marvin’s life depends on an NVIDIA GPU with CUDA 7.5 and cuDNN 3.

https://github.com/PrincetonVision/marvin/

Marvin is a GPU-only neural network framework made with simplicity, hackability, speed, memory consumption, and high dimensional data in mind.

The MIT License (MIT)
Copyright (c) 2015 Princeton Vision Group

Understanding Convolutional Neural Networks for NLP | WildML

http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/

In this post I’ll try to summarize what CNNs are, and how they’re used in NLP.

Code Excited Linear Prediction - Wikipedia

https://ja.wikipedia.org/wiki/Code_Excited_Linear_Prediction

Code Excited Linear Prediction（CELP、セルプ）は、1985年に米AT&Tの M.R. Schroeder と B.S. Atal が提案した音声符号化アルゴリズム。

携帯電話の音声圧縮のベースとなっているアルゴリズムらしい。

[1510.04189] Improving Back-Propagation by Adding an Adversarial Gradient

http://arxiv.org/abs/1510.04189

A simple extension to the back-propagation method is proposed, that adds an adversarial gradient to the training.

ニューラルネットで、1パス目は重みを更新せずに誤差を計算して、2パス目はその誤差を加工した値を入力値に加算して通常の学習を行うというシンプルな手法で精度が上がるというもの。

Algorithm 1 だけでだいたい分かるが、

z = x + ε * sign(e)

で、x は1パス目の入力値、e は1パス目で計算した誤差、sign 関数は { +1, -1 } を返す関数、ε は揺らす大きさ（係数）、z は2パス目の入力値となる。

As stated in Goodfellow et al. (2014) and Fawzi et al. (2015), adding adversarial perturbations is quite different from adding input noise. Adding noise will direct the model to increase the margin in all possible directions around the training samples. A model has limited capacity, and this may limit the achievable margin in the directions that matters most, where the margins are smallest.

モデルの表現力には限界がある。ノイズを加える手法は不要な方向にもマージンを大きくしてしまうのが問題だが、adversarial perturbations はそれとは違うということが記述されている。

[1409.2329] Recurrent Neural Network Regularization

http://arxiv.org/abs/1409.2329

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, ...

LSTM に dropout を適用するには、リカレント（同じ層の1つ前の時間ステップ）の隠れユニットに適用するのではなく、入力（または１つ下の層の隠れユニット）に対して適用するとよいという話。

Figure 2 と p3 の下の式の太字 D の箇所を参照するとよい。