Assessment

1 Assessment overview

This course will be self-assessed on the basis of a Portfolio consisting of a Reflective Journal and a Group Project.¹

The Portfolio is the sum of weekly self-reflections in the Reflective Journal, the feedback you will receive on your reflections, the Group Project and the Final Self-reflection. The course is assessed based on all of these components. There are no weights for the different components, so it is important you engage with all of them.

The following sections give more details on the Reflective Journal and the Group Project.

2 Reflective Journal

Deadlines

• You should post your self-reflection each week after you have attended your classes.

• There isn’t a hard deadline and if necessary you can skip a week and post two reflections the week after.

Learning is an active process: it happens best when we’re all engaged. Learning is also something that we can learn how to do better. Think of a hobby you might have or a skill you’ve learned: you’ve probably perfected it for a while, paying close attention to what did or didn’t work.

That’s why both students and the lecturer (Stefano) will reflect on how the course is going and what they’ve learned. After each class we’ll all write short self-reflections in the Reflective Journal. I’ll also comment on yours and you’ll be able to comment on mine.

Each of us will have our own Reflective Journal on Learn. The tutors and I will be able to see your Reflective Journal, but you won’t be able to see other students’ Journals. My Journal (the lecturer’s) will be visible to all and you will also be able to comment on it.

Here are some example prompts to get you started, but you can really write what you want!

Example prompts

1. Which Notebook entries/other materials did I go through this week?

2. How did I approach this week’s Challenge?

3. What has changed in my thinking about quantitative methods so far (if at all)?

4. How would I describe my participation in class activities? Am I happy with it?

5. In what ways have I grown intellectually so far, if at all?

6. Is there anything I’m particularly proud of?

7. Did I struggle with anything? If so, did I ask for help?

2.1 Instructions for Weekly self-reflections submission

After every class you’ll need to write one self-reflection in the Reflective Journal on Learn. Afterwards you need to read our feedback on your entries.

What happens is:

1. We have class.
2. We reflect on what we’ve learned and what our expectations are for next class.
3. We write down brief reflections (1-paragraph) in one entry in the Reflective Journal on Learn, e.g. one titled “Week 1 Seminar”, “Week 1 Lab”, and so on.
4. I comment on your reflections, and you might comment on mine.
5. I might also give general feedback in a collaborative discussion in class or via Learn announcements.

In sum: after any given class, you’ll write a short self-reflection. Within a reasonable amount of time, I’ll react to these.

3 Group project and Final Reflection

Deadlines

• You must submit a Project Proposal to Turnitin by Thursday 7th November at noon on Learn > Assessment.
• You must submit your Final Reflection, including the product of your Group Project to Turnitin by Thursday 12th December at noon on Learn > Assessment.

3.1 Overview

As part of the assessment you will have to work on a Group Project.

This project can be anything (some ideas below) so the structure of what you will have to submit differs depending on what you end up doing.

Independent of the type of project, you will have to write a Final Reflection on your experience with the course and with working on you Group Project. This reflection is what you submit to Turnitin on Learn by the deadline of Thursday 12th December. The reflection should contain the “product” of your Group Project, either as text following the reflection or as a link (since the Group Project can be any of different types of products).

The requisites for the Group Project are the following:

• It has to be about linguistics, i.e. research on Language and/or languages.

• It has to be on knowledge-oriented (aka “basic”) research, not on application-oriented (aka “applied”) research.

• It has to be in or on R.

Some ideas for Group Projects:

• Pick a linguistic question that interests you and run a mini quantitative study (this could well be a study where you also act as participants).

• Find a linguistic topic and collect published effects from the relevant literature (the first step towards a fully-fledged meta-analysis).

• Design exercises and/or tutorials for the course using linguistic data.

• Write a collection of sonnets on the replicability crisis.

• Write an essay on minoritised statisticians.

• Choreograph an interpretative dance that illustrates how coefficients of a model of lexical decision task data are estimated with the Markov Chain Monte Carlo algorithm in a regression model.

Note that the project has to be on a “knowledge-oriented” topic, rather than on a application of statistics to practical problems or language processing/technology.

So for example the following project is fine: “Do participant respond faster when listening to synthetic speech generated by algorithm A vs those generated by algorithm B?” But the following projects will not be appropriate for the Group Project: “We want to write a speech-to-text algorithm in R” or “We will develop a natural language virtual assistant for online shopping” or “I will train a forced-aligner model for a new language”.

As explained above, you will have to submit a Final Reflection including the product or link to the product of the Group Project by by the deadline of Thursday 12th December at noon.

For any question about assessment, post your question on Piazza (unless it’s of a sensitive nature, then get in touch with Stefano).

Warning

Each project will be different, so if you are unsure about how to approach your project idea or if you need help finding an idea in the first place, please get in touch with Stefano!

Stefano does not have telepathic powers (yet) and the only way for him to help you is for you to reach out. :)

3.2 Instructions for Project proposal submission

Deadline

• You must submit a Project Proposal to Turnitin by Thursday 7th November at noon on Learn > Assessment.

• You will have to submit a very short project proposal for approval by Thursday 7th November to Turnitin on Learn, but the earlier you submit the earlier you can start working on it! We will be checking submissions as they arrive and ask get in touch if there are any issues.
• Only one person per group should submit the proposal (there is no need for everyone in the group to submit the project proposal and you can add all the names of the group members if you wish).
• A few sentences explaining the project will be sufficient. Note that this deadline is an informal deadline and there is no possibility of getting an extension.

The following boxes give you some examples of possible projects and project proposals, with an explanation on how you can go about completing these projects.

Example project 1

Project proposal

We want to replicate the study by Keogh et al. The study use an artificial language learning paradigm to test the relationship between working memory and regularisation. We will run the exact same experiment with 5 participants and conduct the same analysis they did.

How to complete it

This is a replication project. You will need to understand the original study design, make sure you have the same materials and protocols and you will have to recruit 5 participants (these can be students from the QML course, including yourselves). After data collection, you should analyse the data as in the original study and write a short report focussing on the analysis approach and results. Specifically, you should discuss how the new results compare with the ones from the original study.

Example project 2

Project proposal

We want to collect recordings of ourselves reading a passage to illustrate different accents. We will include accents from: Scotland, England, India, Tanzania and Hungary. We will curate the data by formatting it so that it’s FAIR (Findable, Accessible, Interoperable, Reusable) and we will make it available on the Open Science Framework with a CC-BY license. The data will be thoroughly documented.

How to complete it

This is a data curation project. You will collect, format, organise and document data which you will make available online. This requires learning about file data management, file formats, Open Research practices and principles.

Example project 3

Project proposal

We want to draw an abstract paining that represents the analysis approach used in Coretta et al.. The painting will be accompanied by a written piece with an explanation of the approach, our understanding and how this has shaped the painting.

How to complete it

This is creative project. You will have to carefully read a paper, understand the analysis, which will likely be of a more advanced level compared to what you have learnt so far, and use your creativity to create a painting (no need to be artists to make a painting!). The written piece will have to show that you have understood and internalised the analysis and that this understanding has driven the production of the painting. What you submit is the written piece and high resolution photo of the painting.

Example project 4

Project proposal

We are interested in the relationship between vowel height and vowel duration in Hungarian. We will run a small-scale study with recordings from 2 or 3 students who are speakers of Hungarian. We will analyse the vowel duration using Bayesian regression models.

How to complete it

This is research project. You will have to think of a research question/hypothesis (they do not have to be novel and you can reuse existing questions/hypotheses) and collect and analyse data. Note that we are restricted by ethics requirements so you can’t record people that are not UoE students, unless you want to apply for ethics (which you can). The written report will have to focus on the data collection and analysis (you should put less focus on literature review and discussion).

Example project 5

Project proposal

We want to re-analyse the data from [Ota 2013]. The original study used frequentist regression models, but we will instead apply a Bayesian regression model.

How to complete it

This is robustness assessment project. You will have to get hold of the original data and re-analyse it using a different approach than that in the original study. The report will describe the original approach and your approach, report the results and assess how these are different (if at all) from the original study’s results.

3.3 Instructions for Group Project and Final Reflection submission

Deadline

• You must submit your Final Reflection, including the product of your Group Project, to Turnitin by Thursday 12th December at noon on Learn > Assessment.
• Normal extensions are not possible, so carefully plan your time accordingly. Learning Adjustments and Special Circumstances extensions are possible. Note that LA/SC extensions do not extend to group-mates who don’t have LA/SC in place.

• You should submit a PDF file.
• You should use this Quarto document template (right-click and download). For the submission, please render the Quarto file to PDF (I suggest trying this as soon as you download the file, to ensure you have all the necessary software for rendering to PDF).
• Every student should submit. Your submission should include the Group Project output or link to it (we will discard the similarity ratings on Turnitin).
• In your submission, include the Group Project title and the NAMES of the members of the group (not the student number nor the exam number), including yourself, and your proposed mark for yourself (this should be done individually, not at the group level), as per the template linked above.
• If you are unsure about anything, please post a question on Piazza.

4 Feedback and marking

Feedback will be provided to you (1) during class, (2) as comments on your weekly self-reflections and (3) during office hours (it is up to you to book meetings with me and/or the tutors).

This course is self-assessed with co-moderation, which means that at the end of the course you will assign yourself a mark and the mark will be moderated in conversation between you and the Lecturer (Stefano)/Tutors. The conversation will happen either in a meeting or via email:

• If you would like to have a conversation, I have arranged 10-minute online meetings, or “exit interviews”, the week of December 16 (every day, between 1-5pm). You can book these from the usual booking system (link at the bottom of the course homepage).

• If you don’t want to have a conversation, me or the Tutors will get in touch with you via email only in the case that adjustments to the mark have to be made (which in the experience of other colleagues using this approach happens quite rarely!).

Self-assessment with co-moderation

• Based on your learning as evidenced by the Portfolio, you assign yourself a mark (see section on impression-based step marking) when submitting the Final Reflection with the Group Project (by December 12).

• The Course Organiser/Tutors will moderate the self-assigned mark as part of a dialogue with you.

4.1 Impression-based step-marking

To help you thinking about which mark to assign yourself, we will use an “impression-based step-marking” procedure. The procedure works as follows:

• First, think about the mark band you think you deserve.
• Then, choose a step within the band (see below for an explanation).

Once you pick a mark band, you need to pick a step within that band. You can find a description of the bands here: Extended Common Marking Scheme. The following is an Extended Extended Common Marking Scheme made specifically for this course.

Mark (bands)	Grade	Description	Full Description
90-100	A1	Excellent	I engaged with the course contents, classes and the learning process at an exeptional level, beyond any expectation. I have learnt all of the contents of the Notebook plus (possibly advanced) content from other sources. I have practised my R skills above and beyond the methods covered in the Notebook. I feel extremely comfortable finding out how to analyse data in novel contexts.
80-89	A2	Excellent	I engaged with the course contents, classes and the learning process at an outstanding level. I have learnt all of the minimum contents (the mandatory Notebook entries) plus further content from the Notebook and possibly from other sources. I have practised my R skills and I know how to read a variety of data types, create complex plots and model different data types using advanced regression models and possibly other methods. I feel comfortable finding out how to analyse data in contexts that are very dissimilar from the ones seen in class.
70-79	A3	Excellent	I engaged with the course contents, classes and the learning process at an excellent level. I have learnt all of the minimum contents (the mandatory Notebook entries) plus further content from the Notebook. I have practised my R skills and I know how to read a variety of data types, create complex plots and model different data types using regression. I feel comfortable finding out how to analyse data in contexts that are very dissimilar from the ones seen in class.
60-69	B	Very good	I engaged with the course contents, classes and the learning process at a very good level. I have learnt all of the minimum contents (the mandatory Notebook entries) plus some further content covered in the Notebook. I have practised my R skills and I know how to read a variety of data types, create moderately complex plots and model different data types using regression. I feel comfortable finding out how to analyse data in contexts that are dissimilar from the ones seen in class.
50-59	C	Good	I engaged with the course contents, classes and the learning process at a good level. I have learnt all of the minimum contents (the mandatory Notebook entries). I have practised my R skills and I know how to read data, create plots and model data using regression. I might not be able to apply these to contexts that are dissimilar from those seen in the course.
40-49	D	Satisfactory, pass	I minimally engaged with the course contents and classes and the learning process. I have learnt most of the bare minimum (most of the content of the mandatory Notebook entries) but I still have some gaps. I have minimally practised my R skills and I can ran minimal and simplistic quantitative analyses in R, like reading data and creating simple plots.
0-39	E-H	Fail	I did not engage with the course contents and classes nor the learning process at all. I have not learned even the minimum (the content of the mandatory Notebook entries).

Within each mark band, there are three steps: 2, 5 or 8 (for example, in the 60s band there is 62, 65 or 68). The criteria for each of 2, 5, or 8 are the following:

Step	Criteria
2	The criteria for the mark band have been achieved, but there might be minor issues.
5	The criteria for the mark band have been achieved satisfactorily.
8	The criteria for the mark band have been achieved fully.

Footnotes

1. Some of the text on this page is from Itamar Kastner’s Morphology course site.