Jiayi Pan

Jiayi Pan

ę½˜å®¶ę€”

University of Michigan - Ann Arbor

Hi šŸ‘‹

I am Jiayi Pan. A senior student in Computer Science. My research goal to build language agents that operate in physical environments and interact with humans. Towards this goal, my current research focused on language grounding to vision and robotics.

Iā€™m fortunate to have worked with Professors Joyce Chai, Dmitry Berenson, Fan Wu through my undergraduate research.

Outside of work, I love reading, arts, sports on water, implementing random ideas, playing/developing games, being connected with interesting people. Feel free to contact me and have fun via email :)

šŸ“¢: I am applying for PhD programs and Summer Intern for 2023. If you have opportunities, please feel free to contact me via email.

Interests
  • Grounded AI
  • Natural Language Processing
  • Computer Vision
  • Robotics
Education

  • B.S.E in Computer Science, 2021-2023

    University of Michigan, Ann Arbor, U.S.A.

  • B.S.E in Electrical and Computer Engineering, 2019-2023

    Shanghai Jiao Tong University, Shanghai, China

Publications & Manuscripts

šŸ¤« Three of my papers are under anonymous peer review and will be released here once they are accepted šŸ˜‰
* denotes equal contribution
Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification

Jiayi Pan, Glen Chou, Dmitry Berenson. ICRA 2023.

We present a learning-based approach to translate from natural language commands to LTL specifications with very limited human-labeled training data by leveraging Large Language Models. Our model can translate natural language commands at 75% accuracy with about 12 annotations and when given full training data, achieves state-of-the-art performance. We also show how its outputs can be used to plan long-horizon, multi-stage tasks on a 12D quadrotor.
PDF Code
Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification
DANLI: Deliberative Agent for Following Natural Language Instructions

Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Peter Yu, Yuwei Bao, Joyce Chai. EMNLP 2022, Oral.

We propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience. Our deliberative agent achieves 70% improvement over reactive baselines on the challenging TEACh benchmark. Moreover, the underlying reasoning and planning processes, together with our modular framework, offer impressive transparency and explainability to the behaviors of the agent.
PDF Cite Code
DANLI: Deliberative Agent for Following Natural Language Instructions
FairEdit: Preserving Fairness in Graph Neural Networks through Greedy Graph Editing

Donald Loveland, Jiayi Pan, Aaresh Farrokh Bhathena, Yiyang Lu. Manuscript, 2022.

We introduce FairEdit, the unexplored method of edge addition, accompanied by deletion, to promote fairness. FairEdit performs efficient edge editing by leveraging gradient information of a fairness loss to find edges that improve fairness. Our results show that FairEdit performs comparably to many state-of-the-art methods, demonstrating FairEdit's ability to improve fairness across many domains and models.
PDF Cite Code
FairEdit: Preserving Fairness in Graph Neural Networks through Greedy Graph Editing

Side Projects

*
All Deep Learning Robotic Game
Last updated on Sep 28, 2022
A Guided Tour to Neural Radiance Field
A Guided Tour to Neural Radiance Field

A guided tour to neural radiance field, with extensive notes.

Code
Last updated on Oct 28, 2022
Exploiting Language Context and Social Relation for Efficient Federated Learning
Exploiting Language Context and Social Relation for Efficient Federated Learning

A federated learning system that leverages the rich language context and social relation information, achieving highly-competitive performance.

PDF Code Slides
Last updated on Sep 28, 2022
Fairness in Graph Representation Learning
Fairness in Graph Representation Learning

Reimplementation of SOTA algorithms in promoting fairness in Graph Representation Learning

PDF Code
Last updated on Sep 28, 2022
Modular Mars Rover
Modular Mars Rover

A modular-designed remote-controlled robotic car with support of robotic arm, 3 DoF camera, solar panel & power bank, replaceable chassis, disco light & sound.

Code Poster Video
Last updated on Sep 28, 2022
Ray Tracer
Ray Tracer

A ray-tracing based renderer that supports diffuse illumination, specular lighting, shadows, reflection and refractions.

Code
Last updated on Sep 28, 2022
Robot Localization
Robot Localization

Comparison and analysis of Kalman Filter and Particle Filter on a robot localization task in simulated environment.

PDF Code
Last updated on Nov 11, 2022
Shade
Shade

Life in Color, Life in Shadow.

Play
Last updated on Oct 28, 2022
Zeroda
Zeroda

A re-implementation of the original Legend of Zelda game, with a new weapon, scene and many secrets!

Play

Selected Honors/Awards

Tang Jun Yuan JI Scholarship
University of Michigan - Shanghai Jiao Tong University Joint Institute Aug 2021
  • 2 out of 100+ students per year | Highest honor for JI’s Dual Degree students
  • Endowed by Cytus Tang Foundation.
John Wu & Jane Sun Excellence Scholarship
University of Michigan - Shanghai Jiao Tong University Joint Institute Dec 2020
  • 15 out of 1000+ students per year | Highest honor for JI undergraduate students
  • Endowed by Mr John Wu and Mrs. Jane Sun.
Fan Xuji Sholarship
Shanghai Jiao Tong University Nov 2020
  • 10 out of all undergrad students (over 10k) per year | Highest honor for SJTU undergraduate students
  • Endowed by various alumni that were supported by Former University President Fan Xuji.
Chunt-sung Research Scholarhip
Shanghai Jiao Tong University May 2020
  • 50 research projects initiate per year | Top undergraduate research program for SJTU students.
  • Endowed by Tesung-Dao Lee and his wife Hui-Chun Chin.
Second Prize in Chinese Physics Olympiad 2017, 2018
China Association for Science and Technology May 2017
  • Top 100 in Sichuan Province

Back in my highschool days, I was really tired of and bored with all the naive coursework. Just for the fun of it, instead, I spent most of my time learning university-level physics and preparing for the CPhO.

This prize might not look big compared to others listed above, but it is exactly where my passion for science starts. After the “failure” in CPhO, I went on to prepare for the College Entrance Examination (GaoKao) and ranked top 0.1% in 2019.

Now as an AI researcher, I still constantly benefit from the training I had for CPhO. Not only for the domain-specific knowledge I learnt, but also on how to think in the first principle, how to approach, build intuition with, and understand complex systems, etc…

Contact

Schedule:


Direct Contact

  • Email: jiayipan [DOT] umich [DOT] edu