Large Language Models and Transformers

Registration is now open! This thematic semester will take place in Berkeley, California. We encourage interested researchers from across the Canadian AI ecosystem to participate in these workshops (either virtually or in person) to build collaborations and engage in these prescient discussions. A limited number of travel grants are available. Apply for funding as soon as possible.

IVADO invites you to Berkeley, California, to take part in a thematic semester organized in collaboration with the Simons Institute for the Theory of Computing. This program’s overarching goal is to try to understand the ongoing revolution in transformers and large language models (LLMs) through a wide lens. This thematic semester, featuring three collaborative workshops ​​starting in February 2025, will offer a unique opportunity to collaborate and deepen exchanges with world-renowned experts, in a relaxed setting that facilitates discussion, debate, and intellectual cross-pollination.

At a conceptual level, LLMs profoundly change the landscape for theories of human language, of the brain and computation, and of the nature of human intelligence. In linguistics, they provide a new way to think about grammar, semantics, and conceptual representation. In neuroscience, vector models provide a new approach to computational models of the brain. In cognitive science, they challenge our notions of what are the essential elements of human intelligence.

This program will explore very concrete questions about transformers as models of computation. This includes algorithmic ideas to reduce the complexity of training to nearly linear in the length of the input, as well as scaling laws studying how cross-entropy loss scales with model size, data set size, and amount of compute. The program will also explore how scaling laws might help in understanding high-level outcomes such as the emergence of complex skills in LLM models.

At a practical level, it is clear that LLMs will have a profound impact on human society, and issues of alignment, trust, and security will play a central role. Alignment refers to the gap between complex human values and the mechanisms that drive AI decision-making. Related issues include trustworthiness (how do we know the model will do what it’s intended to?), interpretability (can we identify with certainty why a machine learning algorithm delivers a specific answer?), safety (can we safeguard against destructive actions by ML algorithms or humans using them?), security (can we protect data and systems from adversaries?), and fairness (can we safeguard against bias?). The legal and regulatory dimension of technological developments in AI, as well as its practical interaction with the capabilities and design of large language models, will be another key area of inquiry.

This thematic semester is the second part of a special one-year program on large language models and transformers, spanning the 2024-2025 academic year and launched at the Simons Institute for the Theory of Computing.

The three workshops scheduled for spring 2025 will take place on the Simons Institute campus in California, and will highlight the collaborative efforts between the two institutions. As Co-organizer of the Thematic Semester, IVADO is supporting the travel of Canadian-based researchers to attend and participate in the following workshops:

1. LLMs, Cognitive Science, Linguistics, and Neuroscience (Monday, Feb. 3 – Friday, Feb. 7, 2025)
2. The Future of Language Models and Transformers (Monday, March 24 – Friday, March 28, 2025)
3. Safety-guaranteed LLMs (Monday, April 14 – Friday, April 18, 2025)

Please submit the registration form below to register for the thematic semester workshops on “Large Language Models and Transformers” organized in Spring 2025 by IVADO and the Simons Institute for the Theory of computing. These workshops will be held in California, USA.

In person attendance :

IVADO is offering travel grants of $3000 CAD per workshop, with priority given to PhD students, post-doctoral researchers and early-career researchers wishing to travel to California to attend the workshops in person. This initiative aims to support the participation of those who wish to benefit from this opportunity to exchange ideas with renowned experts and other peers in the field.

It is strongly advised to submit your application as soon as possible, as the number of scholarships available is limited. To apply for the travel grant, register for the activities via the form below, then submit a grant application by clicking here.

Please note that in the case of in-person attendance, space will be limited. Seating in our workshops is on a first come, first served basis. In the event the auditorium becomes full, we will open up our overflow room for attendees to view the workshop livestream.

Online attendance :

The workshop playlist URL will be emailed to registered participants. This URL can be used for immediate access to the livestream and recorded talks. Please do not share the playlist URL until talk recordings have been made publicly available on SimonsTV, about 12-15 days after the workshop.

Registration : 

48-72 hours after your registration, you will receive a confirmation e-mail where you can update your registration details at any time for the workshop(s) you have registered for.

Please note: the Simons Institute regularly captures photos and video of activity around the Institute for use in videos, publications, and promotional materials. Content produced as part of this thematic semester could also be used by IVADO.

Questions? Contact IVADO here or Simons Institute Events Staff at simonsevents@berkeley.edu.

If you require specific accommodation, please contact our access coordinator at simonsevents@berkeley.edu with as much advance notice as possible.

IVADO is offering travel grants to attend the proposed workshops, which will be held in California, USA, on the campus of the Simons Institute for the Theory of computing. Priority is given to doctoral students, post-doctoral researchers and early-career researchers who wish to attend one of the three California workshops in person. This opportunity aims to support the engagement of those eager to directly exchange with renowned experts and colleagues in the field.

IVADO will provide a travel grant of $3000 CAD per workshop to cover the cost of travel and accommodations.

The number of travel grants available is limited. To be eligible, we strongly recommend that you register for the workshop of your choice and apply for funding as soon as possible. Funding is granted based on eligibility and relevance of your application, and is available on a first come, first served basis, subject to the availability of funds.

Please note that no funding request submissions will be accepted less than two weeks before the event. Make sure to complete your registration and funding request well before this deadline to maximize your chances of receiving support.

Selection Criteria:

• Active researcher in the field of NLP, LLM, and/or transformers, or a relevant related field;
• Currently employed or enrolled at a Canadian University;
• Doctoral student, postdoctoral researcher or early career researcher (priority criteria, but not excluding).

Stipend Request Form: