dgit.raspbian.org Git - git-annex.git/blob

   1 ### Please describe the problem.
   2
   3 Continuation of the prior brief troubleshooting in [devblog](http://git-annex.branchable.com/devblog/day_518__S3_versioning_finishing_touches/#comment-ced84e6126b523170638f36a73cdb4b1).
   4
   5 Joey:
   6
   7 > As to the problems you're having with that S3 remote, it looks like it does not have public=yes enabled in its configuration at all, which is why git-annex is failing to download from it.
   8
   9 I do not think that is the case.  s3-PUBLIC has `public=yes`.  But it is entirely not clear why git-annex fails to download - no diagnostic error message at --debug level.  Please try yourself:
  10
  11 ### What steps will reproduce the problem?
  12
  13 [[!format sh """
  14 ~datalad > git clone https://github.com/OpenNeuroDatasets/ds001419
  15 Cloning into 'ds001419'...
  16 remote: Enumerating objects: 420, done.
  17 remote: Counting objects: 100% (420/420), done.
  18 remote: Compressing objects: 100% (232/232), done.
  19 remote: Total 420 (delta 99), reused 418 (delta 97), pack-reused 0
  20 Receiving objects: 100% (420/420), 37.43 KiB | 2.88 MiB/s, done.
  21 Resolving deltas: 100% (99/99), done.
  22
  23 ~datalad > cd ds001419
  24
  25 ~datalad/providers/openneuro/ds001419 > git annex enableremote s3-PUBLIC
  26 (merging origin/git-annex into git-annex...)
  27 (recording state in git...)
  28 download failed: Not Found
  29
  30   Remote origin not usable by git-annex; setting annex-ignore
  31 enableremote s3-PUBLIC ok
  32 (recording state in git...)
  33
  34 ~datalad/providers/openneuro/ds001419 > git annex whereis sub-01/anat/sub-01_T1w.nii.gz
  35 whereis sub-01/anat/sub-01_T1w.nii.gz (2 copies)
  36         303581cd-3c58-4a60-b685-42cae4314c68 -- root@1f69c4ed80cf:/datalad/ds001419
  37         c21131c5-9e22-442d-a4d6-22d225603bc9 -- s3-PRIVATE
  38
  39   The following untrusted locations may also have copies:
  40         336309c0-d7f8-430e-bd2b-4bdc61e587cb -- [s3-PUBLIC]
  41 ok
  42
  43 ~datalad/providers/openneuro/ds001419 > git annex get --from s3-PUBLIC --debug sub-01/anat/sub-01_T1w.nii.gz
  44 [2018-09-25 11:30:48.623186962] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
  45 [2018-09-25 11:30:48.6294298] process done ExitSuccess
  46 [2018-09-25 11:30:48.629531355] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
  47 [2018-09-25 11:30:48.63678286] process done ExitSuccess
  48 [2018-09-25 11:30:48.637035079] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..a6e5981635fd80f9186e6aa840312a97153508a4","--pretty=%H","-n1"]
  49 [2018-09-25 11:30:48.648701597] process done ExitSuccess
  50 [2018-09-25 11:30:48.648965608] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
  51 [2018-09-25 11:30:48.649547391] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
  52 [2018-09-25 11:30:48.655933893] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--","sub-01/anat/sub-01_T1w.nii.gz"]
  53 get sub-01/anat/sub-01_T1w.nii.gz (from s3-PUBLIC...) [2018-09-25 11:30:48.662890927] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff-tree","-z","--raw","--no-renames","-l0","-r","4b825dc642cb6eb9a060e54bf8d69288fbee4904","3bbd58c6720ad7a764409c219397a4e15e4ed77d","--"]
  54 [2018-09-25 11:30:48.670609446] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
  55 [2018-09-25 11:30:48.672897086] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
  56 [2018-09-25 11:30:48.681427258] process done ExitSuccess
  57 failed
  58 [2018-09-25 11:30:48.740042089] process done ExitSuccess
  59 [2018-09-25 11:30:48.740263179] process done ExitSuccess
  60 [2018-09-25 11:30:48.740549287] process done ExitSuccess
  61 [2018-09-25 11:30:48.740805353] process done ExitSuccess
  62 git-annex: get: 1 failed
  63
  64 ~datalad/providers/openneuro/ds001419 > git cat-file -p git-annex:remote.log
  65 336309c0-d7f8-430e-bd2b-4bdc61e587cb bucket=openneuro.org datacenter=US encryption=none exporttree=yes fileprefix=ds001419/ host=s3.amazonaws.com name=s3-PUBLIC partsize=1GiB port=80 public=yes publicurl=yes storageclass=STANDARD type=S3 timestamp=1537889429.294083309s
  66 c21131c5-9e22-442d-a4d6-22d225603bc9 bucket=openneuro-private datacenter=US encryption=none exporttree=yes fileprefix=ds001419/ host=s3.amazonaws.com name=s3-PRIVATE partsize=1GiB port=80 storageclass=STANDARD type=S3 timestamp=1531961539.924031958s
  67
  68 ~datalad/providers/openneuro/ds001419 > git annex info
  69 repository mode: indirect
  70 trusted repositories: 0
  71 semitrusted repositories: 5
  72         00000000-0000-0000-0000-000000000001 -- web
  73         00000000-0000-0000-0000-000000000002 -- bittorrent
  74         303581cd-3c58-4a60-b685-42cae4314c68 -- root@1f69c4ed80cf:/datalad/ds001419
  75         3cf65be0-250d-4456-bfee-8501390e28c7 -- yoh@hopa:~/proj/datalad/providers/openneuro/ds001419 [here]
  76         c21131c5-9e22-442d-a4d6-22d225603bc9 -- s3-PRIVATE
  77 untrusted repositories: 1
  78         336309c0-d7f8-430e-bd2b-4bdc61e587cb -- [s3-PUBLIC]
  79 transfers in progress: none
  80 available local disk space: 1.76 gigabytes (+1 megabyte reserved)
  81 local annex keys: 0
  82 local annex size: 0 bytes
  83 annexed files in working tree: 7
  84 size of annexed files in working tree: 183.03 megabytes
  85 bloom filter size: 32 mebibytes (0% full)
  86 backend usage:
  87         MD5E: 7
  88
  89 ~datalad/providers/openneuro/ds001419 > git annex info s3-PUBLIC
  90 uuid: 336309c0-d7f8-430e-bd2b-4bdc61e587cb
  91 description: [s3-PUBLIC]
  92 remote: s3-PUBLIC
  93 trust: untrusted
  94 cost: 200.0
  95 type: S3
  96 creds: not available
  97 bucket: openneuro.org
  98 endpoint: s3.amazonaws.com
  99 port: 80
 100 storage class: STANDARD
 101 partsize: 1.07 gigabytes
 102 public: yes
 103 versioning: no
 104 encryption: none
 105 chunking: none
 106 export: yes
 107 remote annex keys: 27
 108 remote annex size: 183.03 megabytes (+ 20 unknown size)
 109
 110 """]]
 111
 112
 113 may be it is somehow interaction with the fact that the bucket is versioned, although export happened with annex version which didn't support versioning?
 114
 115 It would be nice, before introducing versioning information as you outlined in the [comment](http://git-annex.branchable.com/devblog/day_518__S3_versioning_finishing_touches/#comment-01400aba452e6596e2b6b2967a0b9bc1) to enable access to the bucket at least to download the most recent version(s) of the file(s).
 116
 117 ### What version of git-annex are you using? On what operating system?
 118
 119 6.20180913+git52-gdb1644808-1~ndall+1
 120
 121
 122 [[!meta author=yoh]]
 123 [[!tag projects/datalad]]
 124
 125 > Added an error message when download of an invalid url is attempted
 126 > and also made the url be listed by --debug. [[done]] --[[Joey]]