Sources for the DataLad handbook
datalad-handbook, updated
π₯
2023-02-28 07:15:23


The DataLad handbook :orange_book:
This is a living resource on why and - more importantly - how to use DataLad.
The rendered version is here: http://handbook.datalad.org, and is currently
under initial development.
The handbook is a practical, hands-on crashcourse to learn and experience DataLad.
You do not need to be a programmer, computer scientist, or Linux-crank. If you have never touched your computer's shell before, you will be fine. Regardless of your background and personal use cases for DataLad, the handbook will show you the principles of DataLad, and from chapter 1 onwards you will be using them.
Find more general information about the idea behind the handbook in the poster presented at the 2020 OHBM or dive straight into your DataLad adventure.
Contributing
Contributions in any form - pull requests, issues, content requests/ideas, ... are always welcome.
If you are using the handbook and find that something does not work,
please let us know.
Likewise, if you are using DataLad for your individual project, consider contributing by telling us about your use-case.
You can find out more on how to contribute here,
and a list of all contributors so far below, in CONTRIBUTORS.md, and in .zenodo.json.
Notes for Instructors
The book is the basis for workshops and lectures on DataLad and data management.
The handbook's course repository
among other things contains live casts from the code examples in this book and
slides. It is constantly growing, and everyone is free to use the material under
the license terms below. Contributions and feedback are very welcome.
License
CC-BY-SA: You are free to
- share - copy and redistribute the material in any medium or format
- adapt - remix, transform, and build upon the material for any purpose, even commercially
under the following terms:
1) Attribution β You must give appropriate credit, provide a link to the license, and indicate
if changes were made. You may do so in any reasonable manner, but not in any way that suggests
the licensor endorses you or your use.
2) ShareAlike β If you remix, transform, or build upon the material, you must distribute your
contributions under the same license as the original.
Contributors β¨
Thanks goes to these wonderful people (emoji key):
This project follows the all-contributors specification. Contributions of any kind welcome!
Issues
opened on 2023-02-26 22:08:33 by mslw
The project uses requirements and requirements-devel, the latter of which contains datalad-container and Python packages used in the examples. This commit includes both in installation instructions.
Another one-line change adds consistency in putting make in verbatim.
Side note: I had to play around a bit to make the build happen on my Fedora PC and found out that the best solution is to remove only the run records that my changes touched (and not use the entire clean-build or clean-beyond-basics), otherwise the run would take long and eventually fail due to some installation-specific issues (e.g. credentials, IIRC hcp, in beyond-basics; or containers in basics). In the end, I felt that using the appveyor build may be the best way forward for me. But all this wasn't hard to figure out based on the provided information, so probably no need to expanding on that in the contributing guide.
opened on 2023-02-22 09:45:58 by jsheunis
https://github.com/datalad/datalad-catalog/issues/250
opened on 2023-02-14 15:22:56 by adswa
Came up in the office hour today, might fit in well into the publication chapter.
opened on 2023-01-31 15:40:03 by adswa
https://github.com/datalad/datalad/issues/7274#issuecomment-1410608967
opened on 2022-12-19 15:20:57 by yarikoptic
@asmacdo showed interest to participate in ongoing handbook hackathon and I thought that it might be great to have a use case show case for dandisets (super dataset at https://github.com/dandi/dandisets, individual at https://github.com/dandisets, asyncio code to update those from the archive within the tools/
of dandisets) and https://github.com/datalad/datalad-fuse/ extension. Dandisets are "special" in that typical files are there large but for access to metadata etc, only small portion of data is needed to be accessed. In datalad-fuse we use https://github.com/fsspec/filesystem_spec/ with local caching, to provide efficient sparse access to remote annexed files which have an http* url
associated with them.
In datalad core we had a request for streaming https://github.com/datalad/datalad/issues/4003 -- so it might be useful to highlight how streaming could be implemented, via fsspec interface within datalad-fuse or directly via FUSE filesystem of that one.
WDYT datalad-handbook folks about such a section? (attn @adswa @mih)
opened on 2022-12-14 15:51:33 by lukas-mertens
Is your feature request related to a problem?
At the moment I can't find a description about if anything breaks if I use git annex assistant.
Describe the solution you'd like
It would be nice to have a short paragraph explaining what happens, when I do it, e.g. in a subdataset. It's a really nice feature I would not like to miss, to be able to selectively sync certain files automatically between devices.
One usecase as an example:
I have a subdataset including downloaded youtube-videos. I setup wanted and unwanted-rules in git-annex, so that they are automatically dropped from my pc and wanted on my media-server.
Describe alternatives you've considered
No response
Additional context
No response
Releases
Handbook love 2023-02-14 14:57:24
A handbook release matching (delayed) the release of DataLad v0.18.0.
This release contains notable changes resulting from a Handbook hackathon in December 2022. It also increases the required version of DataLad to 0.17.
With thanks to many old and new contributors: @mih @christian-monch @asmacdo @mslw @jsheunis @Remi-Gau @yarikoptic, happy Valentines day, and happy International Love Data Week!
Better late than never 2022-12-20 15:05:55
Handbook release matching the release of DataLad v0.17.0, which was for the most part only an internal change to a different testing framework (pytest instead of nose), and thus delayed on the handbook side.
It includes contributions the new contributors @Arshitha, @ajveltri22, @complexbrains, @jkosciessa, @candleindark, and @asmacdo, as well as contributions from established contributors @mslw, @sappelhoff, @kimsin98, @adswa, @yarikoptic.
Thank you to everyone contributing :heart:
Spring cleaning! 2022-04-15 13:11:34
Handbook release matching the release of DataLad v0.16.0.
Among the most notable changes, this release contains adjustments to API changes introduced in DataLad 0.16 as well as documentation on newly introduced commands and features. In addition, it adds additional content on contributing, writing DataLad extensions, and publishing to s3 buckets.
The changelog below is autogenerated. Please see this repos CHANGELOG for an executive summary.
What's Changed
- docs: add mslw as a contributor for review by @allcontributors in https://github.com/datalad-handbook/book/pull/763
- Refactor the chapter on publishing by @adswa in https://github.com/datalad-handbook/book/pull/762
- Refactor the chapter on publishing by @adswa in https://github.com/datalad-handbook/book/pull/761
- BF: Fix link by @adswa in https://github.com/datalad-handbook/book/pull/764
- [ENH] replace issue template with the new "form" format by @Remi-Gau in https://github.com/datalad-handbook/book/pull/768
- docs: add Remi-Gau as a contributor for infra by @allcontributors in https://github.com/datalad-handbook/book/pull/769
- [FIX] update css unordered list in important note when viewing by @Remi-Gau in https://github.com/datalad-handbook/book/pull/770
- docs: add Remi-Gau as a contributor for code, design by @allcontributors in https://github.com/datalad-handbook/book/pull/772
- Fix two minor typos by @eort in https://github.com/datalad-handbook/book/pull/775
- add author to zenodo by @eort in https://github.com/datalad-handbook/book/pull/777
- docs: add eort as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/776
- CI: Fix readthedocs build by @adswa in https://github.com/datalad-handbook/book/pull/774
- Fix typo in section Advanced 1.1 by @eort in https://github.com/datalad-handbook/book/pull/778
- Fix a typo in a command by @mslw in https://github.com/datalad-handbook/book/pull/780
- docs: add mslw as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/781
- Computing on clusters: Update chapters with a link to the fairly big workflow by @adswa in https://github.com/datalad-handbook/book/pull/783
- Add a note on how to currently glob for several levels of directories by @adswa in https://github.com/datalad-handbook/book/pull/785
- docs: add AKSoo as a contributor for review by @allcontributors in https://github.com/datalad-handbook/book/pull/786
- New chapter on contributing to datalad by @adswa in https://github.com/datalad-handbook/book/pull/782
- Install: recommend git annex filter-process on windows installations by @adswa in https://github.com/datalad-handbook/book/pull/791
- Maintenance - Speed up sections with imagenette dataset by @adswa in https://github.com/datalad-handbook/book/pull/790
- Mention new recursive glob option in chapter on run by @adswa in https://github.com/datalad-handbook/book/pull/792
- Update Gin as data source workflow by @mslw in https://github.com/datalad-handbook/book/pull/793
- docs: add mslw as a contributor for content by @allcontributors in https://github.com/datalad-handbook/book/pull/795
- Add a section on good practices for creating interoperable file names by @adswa in https://github.com/datalad-handbook/book/pull/794
- Update filenaming.rst - minor typo and url fixes by @mslw in https://github.com/datalad-handbook/book/pull/796
- docs: add Myrskyta as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/798
- BF: Makefile - replace leading spaces with tabs by @yarikoptic in https://github.com/datalad-handbook/book/pull/799
- fix typo by @tguiot in https://github.com/datalad-handbook/book/pull/801
- docs: add tguiot as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/802
- Workflow: Update package index by @adswa in https://github.com/datalad-handbook/book/pull/804
- Link fixes for failing link check by @adswa in https://github.com/datalad-handbook/book/pull/803
- Revert "Links: Remove link to old benchmarks on previous website" by @adswa in https://github.com/datalad-handbook/book/pull/807
- [ENH] Add new 'Create your own extension' section by @jsheunis in https://github.com/datalad-handbook/book/pull/812
- New content: Mention credential integration with Git by @adswa in https://github.com/datalad-handbook/book/pull/814
- Fix/clarify passing git init options though create to change the default branch by @adswa in https://github.com/datalad-handbook/book/pull/815
- DGPA workshop contents by @adswa in https://github.com/datalad-handbook/book/pull/816
- fix wrong figure name by @jhpb7 in https://github.com/datalad-handbook/book/pull/820
- docs: add jhpb7 as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/823
- [ENH] Fix various typos by @jsheunis in https://github.com/datalad-handbook/book/pull/809
- Update dgpa.rst by @eltociear in https://github.com/datalad-handbook/book/pull/825
- docs: add eltociear as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/826
- Release-fixes for the chapter on RIA stores by @adswa in https://github.com/datalad-handbook/book/pull/810
- Remove version pinning of Sphinx by @adswa in https://github.com/datalad-handbook/book/pull/827
New Contributors
- @eort made their first contribution in https://github.com/datalad-handbook/book/pull/775
- @mslw made their first contribution in https://github.com/datalad-handbook/book/pull/780
- @tguiot made their first contribution in https://github.com/datalad-handbook/book/pull/801
- @jhpb7 made their first contribution in https://github.com/datalad-handbook/book/pull/820
- @eltociear made their first contribution in https://github.com/datalad-handbook/book/pull/825
Full Changelog: https://github.com/datalad-handbook/book/compare/v0.15...v0.16
LaTeX improvements 2021-11-25 09:39:47
This release comes 2 months delayed, but belongs to the DataLad 0.15.0
release from September 2021.
The changelog is autogenerated, and hopefully does not contain too many errors.
The majority of work was maintenance, and improvements of the LaTeX backbone of the handbook (huge thanks to @mih for this), rendering the PDF version of the handbook much more beautiful and towards an actually printable book.
What's Changed
- riastores: Avoid deprecated --no-storage-sibling by @kyleam in https://github.com/datalad-handbook/book/pull/641
- Minor changes re code-blocks in PDF by @mih in https://github.com/datalad-handbook/book/pull/647
- Shorten PDF for printed book by @adswa in https://github.com/datalad-handbook/book/pull/648
- Even more adjustments for a printed book by @adswa in https://github.com/datalad-handbook/book/pull/650
- docs: add Remi-Gau as a contributor by @allcontributors in https://github.com/datalad-handbook/book/pull/653
- Python installation instructions by @adswa in https://github.com/datalad-handbook/book/pull/651
- Help: Note about asyncio errors and workaround for Jupyter by @adswa in https://github.com/datalad-handbook/book/pull/646
- More fancy boxes in PDF by @mih in https://github.com/datalad-handbook/book/pull/655
- Further streamlining boxes by @mih in https://github.com/datalad-handbook/book/pull/656
- Numbered, listable and referencable boxes by @mih in https://github.com/datalad-handbook/book/pull/657
- Optional float mode for PDF boxes by @mih in https://github.com/datalad-handbook/book/pull/658
- No vertical lines in a table by @mih in https://github.com/datalad-handbook/book/pull/659
- Bettertables by @mih in https://github.com/datalad-handbook/book/pull/660
- Relax positioning of some boxes, where appropriate by @adswa in https://github.com/datalad-handbook/book/pull/661
- New boxtype
importantnote
by @mih in https://github.com/datalad-handbook/book/pull/662
- Let figures and tables float. by @mih in https://github.com/datalad-handbook/book/pull/663
- Minor change to align title page with print cover by @mih in https://github.com/datalad-handbook/book/pull/665
- Logo update by @mih in https://github.com/datalad-handbook/book/pull/667
- Disentangle too long and nested Findoutmore's by @adswa in https://github.com/datalad-handbook/book/pull/666
- note -> importantnote by @mih in https://github.com/datalad-handbook/book/pull/664
- Tune footnotes by @mih in https://github.com/datalad-handbook/book/pull/669
- TOC fixes by @mih in https://github.com/datalad-handbook/book/pull/670
- Prevent multiple definition of useless labels by @mih in https://github.com/datalad-handbook/book/pull/671
- New cover art for the Intro part by @mih in https://github.com/datalad-handbook/book/pull/672
- Install: Rework macOS installation by @adswa in https://github.com/datalad-handbook/book/pull/675
- docs: add tamaracha as a contributor by @allcontributors in https://github.com/datalad-handbook/book/pull/676
- In-text references of find-out-mores by @mih in https://github.com/datalad-handbook/book/pull/678
- windowsworkarounds -> windows-wit by @mih in https://github.com/datalad-handbook/book/pull/679
- Adjustments for a printed PDF that work also in html by @adswa in https://github.com/datalad-handbook/book/pull/680
- Refurbish installation instructions by @mih in https://github.com/datalad-handbook/book/pull/682
- BF: Fix a mixed up redirection couple by @adswa in https://github.com/datalad-handbook/book/pull/683
- Beautify the PDF further by @mih in https://github.com/datalad-handbook/book/pull/684
- Make term references accessible in the printed handbook by @mih in https://github.com/datalad-handbook/book/pull/685
- Small update of the extension list by @mih in https://github.com/datalad-handbook/book/pull/686
- Remove quotes from config file displays by @bpoldrack in https://github.com/datalad-handbook/book/pull/689
- On cloning openneuro/openfmri data from the datalad superdataset by @adswa in https://github.com/datalad-handbook/book/pull/690
- Print adjustment chapter 3 onwards by @adswa in https://github.com/datalad-handbook/book/pull/691
- Remove forgotten Windows-workarounds by @adswa in https://github.com/datalad-handbook/book/pull/692
- Add a rough sketch of a DataLad intro for Repronim workshop in Yale by @adswa in https://github.com/datalad-handbook/book/pull/693
- Make the index more comprehensive by @mih in https://github.com/datalad-handbook/book/pull/694
- Add redirector for ?cheatsheet by @yarikoptic in https://github.com/datalad-handbook/book/pull/701
- Adds a Github-actions based CI setup by @adswa in https://github.com/datalad-handbook/book/pull/703
- add redirects for chapters mentioned in ukb manuscript by @adswa in https://github.com/datalad-handbook/book/pull/702
- Another sweep for printing by @adswa in https://github.com/datalad-handbook/book/pull/704
- Fix typo by @mih in https://github.com/datalad-handbook/book/pull/705
- Fix ref tag to close #709 by @jsheunis in https://github.com/datalad-handbook/book/pull/710
- docs: add jsheunis as a contributor by @allcontributors in https://github.com/datalad-handbook/book/pull/711
- GIN: On using Gin as an autoenabled special remote by @adswa in https://github.com/datalad-handbook/book/pull/707
- Mention --assume-ready of datalad-run by @adswa in https://github.com/datalad-handbook/book/pull/699
- ENH: Relative links in sidebar by @effigies in https://github.com/datalad-handbook/book/pull/718
- ENH: Use Zenodo record that resolves to latest version by @effigies in https://github.com/datalad-handbook/book/pull/717
- Add FAQ on GitHub displaying git-annex as the default by @adswa in https://github.com/datalad-handbook/book/pull/722
- docs: add jsheunis as a contributor for content, example by @allcontributors in https://github.com/datalad-handbook/book/pull/726
- Windows weird behavior with file extensions by @adswa in https://github.com/datalad-handbook/book/pull/725
- Mention --dry-run by @adswa in https://github.com/datalad-handbook/book/pull/724
- help: On line endings and autocrfl true configurations for windows users by @adswa in https://github.com/datalad-handbook/book/pull/723
- Add walkthrough for using S3 as a special remote by @jsheunis in https://github.com/datalad-handbook/book/pull/721
- docs: add cni-md as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/728
- Fix left-over exit code by @adswa in https://github.com/datalad-handbook/book/pull/727
- Fix inconsistent description of push behavior by @AKSoo in https://github.com/datalad-handbook/book/pull/734
- docs: add AKSoo as a contributor for bug, content by @allcontributors in https://github.com/datalad-handbook/book/pull/735
- Fix typos using codespell by @sappelhoff in https://github.com/datalad-handbook/book/pull/732
- docs: add sappelhoff as a contributor for bug by @allcontributors in https://github.com/datalad-handbook/book/pull/737
- [ENH] add sphinx-copybutton extension by @sappelhoff in https://github.com/datalad-handbook/book/pull/615
- add Stephan as contributor to zenodo file by @jsheunis in https://github.com/datalad-handbook/book/pull/739
- Fix errors in 9.3 back and forth in time by @AKSoo in https://github.com/datalad-handbook/book/pull/740
- MNT: Update outputs to 0.14.4 by @adswa in https://github.com/datalad-handbook/book/pull/742
- Add a monthly cron job to check links in the handbook by @adswa in https://github.com/datalad-handbook/book/pull/743
- Fix broken or permanently redirected links, as discovered by Sphinx's make linkcheck by @adswa in https://github.com/datalad-handbook/book/pull/744
- Update 101-115-symlinks.rst by @lilikapa in https://github.com/datalad-handbook/book/pull/747
- "ultimatively" => "ultimately" by @arokem in https://github.com/datalad-handbook/book/pull/749
- Fixes typo by @arokem in https://github.com/datalad-handbook/book/pull/750
- This is the name of the file that was created earlier on. by @arokem in https://github.com/datalad-handbook/book/pull/751
- "temper" => "tamper" by @arokem in https://github.com/datalad-handbook/book/pull/752
- FIX: GIN siblings - fix typo in code and note that
--url
should not end with .git
by @oesteban in https://github.com/datalad-handbook/book/pull/753
- MAINT: Add @oesteban to zenodo record by @oesteban in https://github.com/datalad-handbook/book/pull/755
- docs: add oesteban as a contributor for data, data, data, bug by @allcontributors in https://github.com/datalad-handbook/book/pull/754
- MNT: Fix link to git-annex docs on GitLab, mention deprecation by @adswa in https://github.com/datalad-handbook/book/pull/756
- Fixlinks by @adswa in https://github.com/datalad-handbook/book/pull/757
- Numbered boxes and references by @mih in https://github.com/datalad-handbook/book/pull/759
New Contributors
- @jsheunis made their first contribution in https://github.com/datalad-handbook/book/pull/710
- @AKSoo made their first contribution in https://github.com/datalad-handbook/book/pull/734
- @oesteban made their first contribution in https://github.com/datalad-handbook/book/pull/753
Full Changelog: https://github.com/datalad-handbook/book/compare/v.14.0...v0.15
We're getting usable on Windows! 2021-02-02 14:53:29
Handbook release matching the release of DataLad v0.14.0.
Like the software release, this handbook release improves the situation on/for Windows systems starkly from what we had before.
With contributions from Tristan Glatard, Ariel Rokem, Remi Gau, Surya Teja Togaru, Judith Bomba, Konrad Hinsen, Wu Jianxiao, MaΕgorzata Wierzba, Stefan Appelhoff, and Michael Joseph -- thank you!
Refactoring or deletions
- Overhaul Windows installation instructions #588
- Adjustments for GitHub's user-password deprecation #626, #592
Additions
Introduction
- git-annex installations with custom built git-annex with MagicMime support #603
- A quick-start guide for OpenNeuro #585
Basics
- Disambiguation on configurations #627 with thanks to John Lee for the issue at datalad
- A new section on how to debug and troubleshoot problems - with thanks to Tristan Glatard for the idea and contributions #538
Advanced
- A chapter on large-scale fair processing with parallel datalad-run calls #591
- A new section on configuring subdataset clone candidates and their priority #548
- A new chapter/section that compares the tool DVC to DataLad #569
Usecases
- Addition of a machine-learning application with DataLad #581
- Addition on Human Connectome Project (HCP) AWS credentials (thanks to Michael Joseph) #622
- Addition of a hands-on tutorial for reproducible papers #608, with thanks to MaΕgorzata Wierzba for feedback and contributions
Miscellaneous additions
- A variety of code lists and introductions #630, #613
- A few new permalinks: git-lfs #624, MPIB intro #614
- A new expandable section "Windows workaround" for Windows-specific notes and explanations #532
- Large amount of Windows adjustments in the Basics #588
- FAQs on copying locked files out of datasets, and on caveats with the BIDS validator - with thanks to Remi Gau #570, #562
- The handbook's GitHub repository received a welcome bot (with thanks to The Turing Way project for CC-BY illustrations), and a "Discussions" Forum
- The handbook's frontpage links to the cheat sheet with a nice illustration #578
DataLad Handbook v0.13 2020-06-23 19:26:44
For zenodo archival