A Cross-Sectional Study of the Completeness of Preregistrations by Psychological Authors From German-Speaking Institutions #MetaScience
https://journals.sagepub.com/doi/10.1177/25152459251357568
Slightly more than 50% of all preregs (53.42%) include all elements of the proposed completeness index.
A look at the 1/
#metascience
3 posts3 participants0 posts today
The (Non)Academic Community Forming around Replications: Mapping the International Open Science space via its Replication Initiatives https://osf.io/preprints/socarxiv/rbyt6_v2
"The OS reform movement – and particularly the associated replication community – is anchored primarily in the present, commonly operating with historical and methodological amnesia"
osf.ioOSF
Following a wonderful workshop at FSCI 2025, the @force11 team is organizing a PREreview Club to review #metascience #preprints.
They are currently looking for participants to commit at various levels of engagement to help start up the club, especially people willing to help recruit people to fill roles for each review, and people who can commit to roles, such as notetaker or article selection.
If that sounds like something you'd be interested in, read more here: https://force11.org/post/organizing-a-force11-prereview-club/
Extraordinary claims require extraordinary faith, whether or not they require extraordinary proof.
Metascience Is More Important Now Than Ever https://undark.org/2025/07/31/opinion-metascience-essential/ #MetaScience This is most likely true, the more difficult question how the field should move forward in terms of culture, approaches, focus etc.
Undark Magazine · Metascience Is More Important Now Than EverOpinion | A growing research field known as ‘the science of science’ will be essential for navigating an uncertain future.
New paper alert! #statistics #metascience "On the poor statistical properties of the P-curve meta-analytic procedure" in JASA. https://raw.githubusercontent.com/richarddmorey/MoreyDavisStober_pcurveASA/refs/heads/main/text/asa_article/Morey_Davis-Stober_2025_JASA_with_supplement.pdf
We show that the "P curve" meta-analysis tests have terrible statistical properties, in spite of being used for over a decade to tell "bad" science from "good". The initial tests should never have made it through peer review. They suffer from extreme sensitivity, arbirary conclusions, inadmissibility, nonmonotonicity in the evidence, and inconsistency in estimation. We recommend they not be used, and that better vetting is needed for methods in metascience.
Journal link: https://www.tandfonline.com/doi/full/10.1080/01621459.2025.2544397
I'm looking for someone who'd be interested in co-hosting a new metascience podcast I'm currently developing.
Also editing/producing roles if people would like to be involved. I'm looking for funding possibilities (if anyone knows any please also get in touch).
Happy to chat and share more for those interested. Ideally looking for someone active in metascience but not a strict requirement
Could AI slow progress in science?
https://open.substack.com/pub/aisnakeoil/p/could-ai-slow-science?
what is progress in science? more papers, but fewer breakthroughs? (how do you measure “progress in science” or recognize a breakthrough?)
the low hanging fruit is picked? but progress should give is taller ladders. also, some progress is in the form of entirely new fields, new trees with new low hanging fruit.
nice bit about better pattern matching and better fits for models needs a better theory to justify the model — geocentric orbits with epicycles made better predictions than heliocentrism until Kepler realized orbits weren’t circular, but elliptic — heliocentrism progressed mostly because it was simpler than all those epicycles.
the gold is at the end of the paper, inspired by an essay by a mathematician named Thurston, who notes that the goal of mathematics is not proofs of theorems but human understanding. to what extent does a result from an AI circumvent human understanding?
AI Snake Oil · Could AI slow science?
Defending science in public we often talk about 'peer reviewed science'. But could this framing contribute to undermining trust in science and holding us back from improving the scientific process? How about instead we talk about the work that has received the most thorough and transparent scrutiny?
Peer review goes a step towards this in having a couple of people scrutinise the work, but there are limits on how thorough it can be and in most journals it's not transparent. Switching the framing to transparent scrutiny allows us to experiment with other models with a path to improvement.
For example, making review open to all, ongoing, and all reviews published improves this. When authors make their raw data and code open, it improves this.
It also gives us a way to criticise problematic organisations that formally do peer review but add little value (e.g. predatory journals). If their reviews are not open and observably of poor quality, then they are less 'thoroughly transparent'.
So with this framing the existence of 'peer reviewed' but clearly poor quality work doesn't undermine trust in science as a whole because we don't pin our meaning and value on an exploitable binary measure of 'peer reviewed'.
It also offers a hopeful way forward because it shows us how we can improve, and every step towards this becomes meaningful. If all we have is binary 'peer reviewed' or not, why spend more effort doing it better?
In summary, I think this new framing would be better for science, both in terms of the public perception of it, and for us as scientists.
“…the sight of a high-ranking university representative stepping in to save a big tech executive from answering a difficult question was deeply embarrassing (or at least should have been) for all concerned.”
And this is the point. #BigTech shielded by academics (a patchy cover at best, though a telling of the power of wealth and its financial control of scientific research). I this case it was about #Google’s AI products where its representative executive Anna Koivuniemi of Google #DeepMind was shielding that the high-ranking university representative and chair, Geraint Rees, ruled an ‘embarrassing’ question out of order.
It’s a short, but enlightening, read well worth your time if your interests include the use of #GenAI in scientific #research.
#ResearchIntegrity #GenAI&ScientificResearch #MetaScience
Read more: https://warrenpearce.pika.page/posts/what-is-metascience-issues-inclusion-and-future-public-value
Warren's blog: Publics, politics, science and technologyWhat *is* metascience? Issues, inclusion and future public value - Warren's blog: Publics, politics, science and technologyThis week, the biennial Metascience Conference came to London’s “Knowledge Quarter” with around 800 participants from a wide range of roles, including researchers, librarians, research funders, publishers, media and corporations. For...
Continued thread
In my own WiP report #VAR4LCR, I presented a mini #metascience study on "Register and task effects in the International Journal of Learner Corpus Research". The audience proposed extending the study to #SLA articles, register studies, and/or book (chapter) publications.
Continued thread
Opening the second day of the First Workshop on #Replication (#WoReLa1), we are treated to a keynote about "Research on Bilingualism as Discovery Science" by Anne Beatty-Martínez. Anne has lots of food for thought for us all. #linguistics #metascience
JOB : Join RoRI as a Research Fellow/Senior Research Fellow in Metascience
https://researchonresearch.org/join-rori-as-a-research-fellow-senior-research-fellow-in-metascience/
Closing 9th of July, based in London/hybrid. Full time, fixed term until 31/12/2027
(apologies for late notice, full disclosure: I am a @RoRInstitute research fellow)
--> please boost for reach <--
Research on Research · Join RoRI as a Research Fellow/Senior Research Fellow in Metascience - Research on ResearchHelp shape the future of research systems, cultures, and decision-making
Continued thread
Okay, Ana Rita Sá-Leite's talk #WoReLa1 genuinely took us (the audience) on an emotional journey: the results of her work point to a meta-analysis highly influenced by a handful of studies from the same with very small sample sizes and very large effects, Type S error, interpreting absence of evidence as evidence of absence, and more... Find out more: https://doi.org/10.1016/j.cognition.2022.105060 #MetaScience #linguistics
doi.orgThe mechanisms underlying grammatical gender selection in language production: A meta-analysis of the gender congruency effectGrammatical gender retrieval during language production has been largely addressed through the picture-word interference (PWI) paradigm, with the aim …
Continued thread
Bernhard Angele is now presenting "Living meta-analyses in Language Sciences" at #WoReLa1. Having explaining the need for such living meta-analyses in a very houmous way, Bernhard demonstrated this very cool project. The main output is a #Shiny app that can perform Bayesian meta-analyses with lots of opportunities to control various parameters AND allows you to upload additional data to an existing meta-analysis. The app can be used as is or the code adapted to your needs: https://dallbrit.shinyapps.io/Breathing_Life_into_MetaAnalysis/. Also check out the associated paper: https://doi.org/10.5334/joc.389 #MetaScience
Are you at #Metascience2025? Come and chat with me during lunch and the poster session this evening about all things open, preprint peer review!
You can also find my poster on Zenodo here: https://doi.org/10.5281/zenodo.15776238
@prereview #peerreview #metascience #inclusivityinpeerreview #traininginpeerreview
Extra excited for #Metascience2025 - 2023 was amazing (photo for reference) and it's my first large conf since finishing the #PhD and joining @TUEindhoven as the prpject lead for our Recognition and Rewards #openscience project.
If you're attending or around London in the coming days, feel free to reach out - I'd love to meet!
Could a novelty indicator improve science?
https://www.nature.com/articles/d41586-025-01882-7 #MetaScience I have mixed feelings about this. The competition for machine-based indicators that align well with human assessments is well-designed, and I agree that researching the role of novelty is interesting. 1/ @openscience
www.nature.comCould a novelty indicator improve science?A competition to develop computational approaches to detect ‘novelty’ in published papers will help metascientists to study how out-of-the-box research changes the scientific landscape.
You are all warmly invited to our #metascience2025 virtual symposium on Friday 27 June 15:00–16:30 CEST on doing #metascience and #interdisciplinary research as an early-career researcher.
Convenor: Anna Leung (psycholinguist and language teacher; University Hospital, LMU, Germany)
Discussants:
- Shawn Hemelstrand (psychologist and methodologist; The Chinese University of Hong Kong)
- Daniel Kristanto (engineer turned neuroscientist; Carl von Ossietzky Universität Oldenburg)
- Elen Le Foll (corpus linguist and language teaching; University of Cologne)
Abstract: https://nomadit.co.uk/conference/metascience2025/p/16480.
Registration is free: https://cos-io.zoom.us/webinar/register/WN_B5N34T4_S8qwj7t_jhfwKA#/registration.
We look forward to discussing these important topics with you! #ECR #PhD #PostDoc #Academia #OpenScience
🄱🄾🄾🅂🅃🄸🄽🄶 🅃🄷🄸🅂 🅃🄾🄾🅃 = 1% 🄼🄾🅁🄴 🄴🄲🅁 🄴🄼🄿🄾🅆🄴🅁🄼🄴🄽🅃!
The fruits of following my curiosity yesterday - a little Sunday Study on citations.
https://tlohde.com/blog/2025/06/becoming-irrelevant/
Motivated by @sundogplanets 's post on citing old papers: https://mastodon.social/@sundogplanets/114677909402696290
I haven't checked, because I wanted to do it anyway, but I'm sure someone out there has done this more thoroughly and properly.