This document summarizes recent research on applying self-attention mechanisms from Transformers to domains other than language, such as computer vision. It discusses models that use self-attention for images, including ViT, DeiT, and T2T, which apply Transformers to divided image patches. It also covers more general attention modules like the Perceiver that aims to be domain-agnostic. Finally, it discusses work on transferring pretrained language Transformers to other modalities through frozen weights, showing they can function as universal computation engines.
This document summarizes recent research on applying self-attention mechanisms from Transformers to domains other than language, such as computer vision. It discusses models that use self-attention for images, including ViT, DeiT, and T2T, which apply Transformers to divided image patches. It also covers more general attention modules like the Perceiver that aims to be domain-agnostic. Finally, it discusses work on transferring pretrained language Transformers to other modalities through frozen weights, showing they can function as universal computation engines.
PFN福田圭祐による東大大学院「融合情報学特別講義Ⅲ」(2022年10月19日)の講義資料です。
?Introduction to Preferred Networks
?Our developments to date
?Our research & platform
?Simulation ? AI
PFN Summer Internship 2021 / Kohei Shinohara: Charge Transfer Modeling in Neu...Preferred Networks
?
The document discusses techniques for modeling charge transfer in neural network potentials (NNPs) for materials simulation. It presents a graph neural network (GNN) baseline architecture called NequIP that predicts short-range atomic energies. Additional techniques are explored to model long-range electrostatic interactions, including adding an electrostatic correction term (Eele) using Ewald summation and using charge equilibration (Qeq) to predict atomic charges. Results show that while Qeq improves charge prediction accuracy, the baseline GNN achieves comparable or better overall accuracy in most datasets tested, possibly because the GNN can already learn electrostatic effects. The document also discusses PyTorch implementations of Ewald summation and Qeq for efficient evaluation.