Now that Google deprecated the Photos<->Drive synchronization, I need another way to back up my photos locally. This program downloads all photos from your Google Photos account and organizes them locally. It is not very user friendly yet, but definitely usable

dermesser, updated 🕥 2022-12-08 07:43:38

photosync

Now that Google deprecated the Photos<->Drive synchronization, I need another way to back up my photos locally. This program downloads all photos from your Google Photos account and organizes them locally. It is not very user friendly yet, but definitely usable.

photosync only ever downloads photos, i.e. the synchronization works from Google Photos as Source of Truth to your local storage. Don't worry about deleting photos locally; although you have to use the slow --resync option to re-download them.

photosync is fast enough for reasonably large libraries. My library of ~50'000 photos was indexed in roughly an hour; the (resumable) download takes another few hours, depending on latency and photo size.

Pull requests are welcome!

Behavior

By default, photosync will ask for OAuth2 authorization on the console, and then immediately start downloading metadata from Google Photos. Once no more new photos are fetched and all metadata is stored in sync.db, photosync will look for photos that are not yet marked as downloaded in the database and fetch the actual image files. By default, it will organize photos in directories like year/month/day/ (numerically, 0-padded), but you can write your own method of mapping photos to directories and override it by setting the path_mapper argument in the Driver constructor called from Main.main().

Note that this (obviously) takes a while for large libraries. But you can always stop photosync and restart it later; without the --all option, it will resume synchronization where it left off.

Albums are currently ignored. Videos are downloaded just like photos.

Install & Use

First, acquire a client secret. This is necessary because this is an open source project, and I don't want client credentials associated with my account floating around in the wild. Also, the daily limit for Photos API calls is at 10'000, so it wouldn't work for a nontrivial number of users anyway.

For this,

go to https://console.developers.google.com.
Ensure you are on the right project or create a new one
Go to the APIs page and enable the Google Photos API.
Set up the OAuth consent screen (otherwise Google will nag you during credentials creation to do it).
Then go to the Credentials page and create a new client ID (type other). Download the JSON file using the download button at the right hand side.
Save the downloaded JSON file and put it somewhere, for example in your photos directory. Pass the path to the file to photosync using the --creds argument. By default, photosync will look for a file called clientsecret.json in the current directory.
After the first run, credentials are cached in the internal SQLite database, meaning you don't have to explicitly specify them on any further invocations.

Once you have gone through the hassle of obtaining the client secret, you can start downloading your photos.

Clone this repository to a convenient place: git clone https://github.com/dermesser/photosync or hg clone git+https://github.com/dermesser/photosync.
Go into the photosync repository and run pip[3] install [--user] .. This installs dependencies needed by the program.
Or you use pipenv using the provided Pipfile: pipenv shell && pipenv install, after which you can use python photosync.py.
Alternatively, build an egg python setup.py bdist_egg and install it easy_install dist/photosync-[version]-py[version].egg
Run it: python3 photosync.py --help or, if you installed it with pip, photosync.py --help.

Consult the help text printed by the last command. Usually you will need to set --dir so that your photos don't end up in the current directory. Typically you would initially run

$ python3 photosync.py --dir=/target/directory --creds=/path/to/clientsecret.json

which also asks you for OAuth authorization. After having uploaded photos (careful: Google Photos exposes new media only a few minutes up to half an hour after uploading!) you can run the following command, which looks for photos and videos that are either older than the oldest known item or newer than the newest known item (obviously missing any that have been uploaded with a date between the oldest and newest items: see --dates below how to fix it):

$ python3 photosync.py --dir=/target/directory

If it turns out you are missing some items locally, you can check Google Photos again for them:

```

to check all photos available for January 2000:

$ python3 photosync.py --dir=/target/directory --dates=2000-01-01:2000-02-01

to check all photos from ever:

$ python3 photosync.py --dir=/target/directory --all ```

Troubleshooting

I have seen Invalid media item ID. errors for valid-looking media item IDs. This happened to a handful of photos, all from the same day. The media item IDs all started with the same prefix which was different than the shared prefix of all other media item IDs (all IDs from one account usually start with the same 4-6 characters). I'm not sure why the API at one point returned those.
To clean this up, remove the invalid IDs from the database (sqlite3 sync.db "DELETE FROM items WHERE id LIKE 'wrongprefix%'") after checking that only a small number of items has this kind of ID (sqlite3 sync.db "SELECT * FROM items WHERE id LIKE 'wrongprefix%'").
Re-fetch metadata for the affected days: python3 photosync.py --dir=.../directory --all --dates=2012-12-12:2012-12-14 (for example)

Issues

Bump certifi from 2020.12.5 to 2022.12.7

opened on 2022-12-08 07:43:37 by dependabot[bot]

Bumps certifi from 2020.12.5 to 2022.12.7.

Commits

9e9e840 2022.12.07
b81bdb2 2022.09.24
939a28f 2022.09.14
aca828a 2022.06.15.2
de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...
b8eb5e9 2022.06.15.1
47fb7ab Fix deprecation warning on Python 3.11 (#199)
b0b48e0 fixes #198 -- update link in license
9d514b4 2022.06.15
4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/dermesser/photosync/network/alerts).

Bump protobuf from 3.15.8 to 3.18.3

opened on 2022-09-23 22:53:08 by dependabot[bot]

Bumps protobuf from 3.15.8 to 3.18.3.

Release notes

Sourced from protobuf's releases.

Protocol Buffers v3.18.3

C++

Reduce memory consumption of MessageSet parsing

This release addresses a Security Advisory for C++ and Python users

Protocol Buffers v3.16.1

Java

Improve performance characteristics of UnknownFieldSet parsing (#9371)

Protocol Buffers v3.18.2

Java

Improve performance characteristics of UnknownFieldSet parsing (#9371)

Protocol Buffers v3.18.1

Python

Update setup.py to reflect that we now require at least Python 3.5 (#8989)

Performance fix for DynamicMessage: force GetRaw() to be inlined (#9023)

Ruby

Update ruby_generator.cc to allow proto2 imports in proto3 (#9003)

Protocol Buffers v3.18.0

C++

Fix warnings raised by clang 11 (#8664)

Make StringPiece constructible from std::string_view (#8707)

Add missing capability attributes for LLVM 12 (#8714)

Stop using std::iterator (deprecated in C++17). (#8741)

Move field_access_listener from libprotobuf-lite to libprotobuf (#8775)

Fix #7047 Safely handle setlocale (#8735)

Remove deprecated version of SetTotalBytesLimit() (#8794)

Support arena allocation of google::protobuf::AnyMetadata (#8758)

Fix undefined symbol error around SharedCtor() (#8827)

Fix default value of enum(int) in json_util with proto2 (#8835)

Better Smaller ByteSizeLong

Introduce event filters for inject_field_listener_events

Reduce memory usage of DescriptorPool

For lazy fields copy serialized form when allowed.

Re-introduce the InlinedStringField class

v2 access listener

Reduce padding in the proto's ExtensionRegistry map.

GetExtension performance optimizations

Make tracker a static variable rather than call static functions

Support extensions in field access listener

Annotate MergeFrom for field access listener

Fix incomplete types for field access listener

Add map_entry/new_map_entry to SpecificField in MessageDifferencer. They record the map items which are different in MessageDifferencer's reporter.

Reduce binary size due to fieldless proto messages

TextFormat: ParseInfoTree supports getting field end location in addition to start.

... (truncated)

Commits

a902b39 No-op whitespace change
ae62acd Updating version.json and repo version numbers to: 18.3
f43ac49 Merge pull request #10542 from deannagarcia/3.18.x
9efdf55 Add missing includes
d1635e1 Apply patch
5b37c91 Update version.json with "lts": true (#10534)
c39d622 Merge pull request #10529 from protocolbuffers/deannagarcia-patch-5
f77d3b6 Update version.json
8178b06 Merge pull request #10503 from deannagarcia/3.18.x
24ca839 Add version file
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump ujson from 4.0.2 to 5.4.0

opened on 2022-07-05 21:35:48 by dependabot[bot]

Bumps ujson from 4.0.2 to 5.4.0.

Release notes

Sourced from ujson's releases.

5.4.0

Added

Add support for arbitrary size integers (#548) @JustAnotherArchivist

Fixed

CVE-2022-31116:

Replace wchar_t string decoding implementation with a uint32_t-based one (#555) @JustAnotherArchivist

Fix handling of surrogates on decoding (#550) @JustAnotherArchivist

CVE-2022-31117: Potential double free of buffer during string decoding @JustAnotherArchivist

Fix memory leak on encoding errors when the buffer was resized (#549) @JustAnotherArchivist

Integer parsing: always detect overflows (#544) @NaN-git

Fix handling of surrogates on encoding (#530) @JustAnotherArchivist

5.3.0

Added

Test Python 3.11 beta (#539) @hugovk

Changed

Benchmark refactor - argparse CLI (#533) @Erotemic

Fixed

Fix segmentation faults when errors occur while handling unserialisable objects (#531) @JustAnotherArchivist

Fix segmentation fault when an exception is raised while converting a dict key to a string (#526) @JustAnotherArchivist

Fix memory leak dumping on non-string dict keys (#521) @JustAnotherArchivist

Fix ref counting on repeated default function calls (#524) @JustAnotherArchivist

Remove redundant wheel dependency from pyproject.toml (#535) @hugovk

5.2.0

Added

Support parsing NaN, Infinity and -Infinity (#514) @Erotemic

Support dynamically linking against system double-conversion library (#508) @musicinmybrain

Add env var to control stripping debug info (#507) @musicinmybrain

Add JSONDecodeError (#498) @JustAnotherArchivist

Fixed

Fix buffer overflows (CVE-2021-45958) (#519) @JustAnotherArchivist

Upgrade Black to fix Click (#515) @hugovk

simplify exception handling on integer overflow (#510) @RouquinBlanc

Remove dead code that used to handle the separate int type in Python 2 (#509) @JustAnotherArchivist

Fix exceptions on encoding list or dict elements and non-overflow errors on int handling getting silenced (#505) @JustAnotherArchivist

5.1.0

Changed

... (truncated)

Commits

9c20de0 Merge pull request from GHSA-fm67-cv37-96ff
b21da40 Fix double free on string decoding if realloc fails
67ec071 Merge pull request #555 from JustAnotherArchivist/fix-decode-surrogates-2
bc7bdff Replace wchar_t string decoding implementation with a uint32_t-based one
cc70119 Merge pull request #548 from JustAnotherArchivist/arbitrary-ints
4b5cccc Merge pull request #553 from bwoodsend/pypy-ci
abe26fc Merge pull request #551 from bwoodsend/bye-bye-travis
3efb5cc Delete old TravisCI workflow and references.
404de1a xfail test_decode_surrogate_characters() on Windows PyPy.
f7e66dc Switch to musl docker base images.
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Authorise application on different computer | Eg: --noauth_local_webserver

opened on 2021-10-26 13:14:46 by vexersa

Thanks for the solution.

It would be excellent to be able to authorise the API connection using something like the common --noauth_local_webserver parameter.

I have a development linux container and would need to authorise on another machine in order to view the browser OAuth process.

🙏

Bump urllib3 from 1.26.4 to 1.26.5

opened on 2021-06-01 23:56:47 by dependabot[bot]

Bumps urllib3 from 1.26.4 to 1.26.5.

Release notes

Sourced from urllib3's releases.

1.26.5

:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

Fixed deprecation warnings emitted in Python 3.10.

Updated vendored six library to 1.16.0.

Improved performance of URL parser when splitting the authority component.

If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors

Changelog

Sourced from urllib3's changelog.

1.26.5 (2021-05-26)

Fixed deprecation warnings emitted in Python 3.10.

Updated vendored six library to 1.16.0.

Improved performance of URL parser when splitting the authority component.

Commits

d161647 Release 1.26.5
2d4a3fe Improve performance of sub-authority splitting in URL
2698537 Update vendored six to 1.16.0
07bed79 Fix deprecation warnings for Python 3.10 ssl module
d725a9b Add Python 3.10 to GitHub Actions
339ad34 Use pytest==6.2.4 on Python 3.10+
f271c9c Apply latest Black formatting
1884878 [1.26] Properly proxy EOF on the SSLTransport test suite
See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Lewin Bormann

recovering Site Reliability Engineer / Physicist-in-training

GitHub Repository

google-photos photos-drive-synchronization google-drive