Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NetCDF: Unknown file format #148

Open
ggebbie opened this issue Feb 20, 2024 · 6 comments · Fixed by #152
Open

NetCDF: Unknown file format #148

ggebbie opened this issue Feb 20, 2024 · 6 comments · Fixed by #152
Labels
bug Something isn't working

Comments

@ggebbie
Copy link
Owner

ggebbie commented Feb 20, 2024 

A
nordic: Error During Test at /home/runner/work/TMI.jl/TMI.jl/test/test_nordic.jl:1
  Got exception outside of a @test
  NetCDF error code -51:
  	NetCDF: Unknown file format
  Stacktrace:
    [1] check
      @ ~/.julia/packages/NetCDF/7hOe9/src/netcdf_helpers.jl:22 [inlined]
    [2] nc_open(ath::String, mode::UInt16, cidp::Vector{Int32})
      @ NetCDF ~/.julia/packages/NetCDF/7hOe9/src/netcdf_c.jl:55
    [3] nc_open
      @ ~/.julia/packages/NetCDF/7hOe9/src/netcdf_helpers.jl:103 [inlined]
    [4] open(fil::String; mode::UInt16, readdimvar::Bool, add_finalizer::Bool)
      @ NetCDF ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:913
    [5] open
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:910 [inlined]
    [6] open(f::NetCDF.var"#55#56"{String}, args::String; kwargs::@Kwargs{})
      @ NetCDF ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:999
    [7] open
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:998 [inlined]
    [8] #ncread#54
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:1052 [inlined]
    [9] ncread
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:1046 [inlined]
   [10] watermassmatrix(file::String)
      @ TMI ~/work/TMI.jl/TMI.jl/src/config.jl:272
@ggebbie ggebbie added the bug Something isn't working label Feb 20, 2024
@ggebbie
Copy link
Owner Author

ggebbie commented Feb 26, 2024

Both google drive and onedrive do not work for large files because of virus scanner. Need alternative hosting method.

@ggebbie ggebbie linked a pull request Mar 7, 2024 that will close this issue
Downloads.download to get HTTPs nordic link #152
Merged
@b-r-hamilton b-r-hamilton reopened this Sep 4, 2024
@b-r-hamilton
Copy link
Collaborator

Hi Jake - I'm getting this issue again for the 2deg x 2deg file. I just double-checked by downloading a fresh copy of TMI, and then running config_from_nc(versionlist()[2]) and I get the following error

julia> config_from_nc(versionlist()[2])
workaround for 2° x 2°
--2024-09-04 12:42:41--  https://docs.google.com/uc?export=download&confirm=&id=1-YEkB_YeQGqPRH6kauhBb2bi_BjVGt9b
Resolving docs.google.com (docs.google.com)... 142.250.65.174, 2607:f8b0:4006:80e::200e
Connecting to docs.google.com (docs.google.com)|142.250.65.174|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://drive.usercontent.google.com/download?id=1-YEkB_YeQGqPRH6kauhBb2bi_BjVGt9b&export=download [following]
--2024-09-04 12:42:41--  https://drive.usercontent.google.com/download?id=1-YEkB_YeQGqPRH6kauhBb2bi_BjVGt9b&export=download
Resolving drive.usercontent.google.com (drive.usercontent.google.com)... 142.250.80.1, 2607:f8b0:4006:80a::2001
Connecting to drive.usercontent.google.com (drive.usercontent.google.com)|142.250.80.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2445 (2.4K) [text/html]
Saving to: ‘TMI_modern_180x90x33_GH11_GH12.nc’

TMI_modern_180x90x3 100%[===================>]   2.39K  --.-KB/s    in 0s      

2024-09-04 12:42:41 (24.3 MB/s) - ‘TMI_modern_180x90x33_GH11_GH12.nc’ saved [2445/2445]

A
ERROR: NetCDF error code -51:
	NetCDF: Unknown file format
Stacktrace:
  [1] check
    @ ~/.julia/packages/NetCDF/49usH/src/netcdf_helpers.jl:22 [inlined]
  [2] nc_open(ath::String, mode::UInt16, cidp::Vector{Int32})
    @ NetCDF ~/.julia/packages/NetCDF/49usH/src/netcdf_c.jl:55
  [3] nc_open
    @ ~/.julia/packages/NetCDF/49usH/src/netcdf_helpers.jl:103 [inlined]
  [4] open(fil::String; mode::UInt16, readdimvar::Bool, add_finalizer::Bool)
    @ NetCDF ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:912
  [5] open
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:909 [inlined]
  [6] open(f::NetCDF.var"#55#56"{String}, args::String; kwargs::@Kwargs{})
    @ NetCDF ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:998
  [7] open
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:997 [inlined]
  [8] #ncread#54
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:1051 [inlined]
  [9] ncread
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:1045 [inlined]
 [10] watermassmatrix(file::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:275
 [11] macro expansion
    @ ./timing.jl:279 [inlined]
 [12] config_from_nc(TMIversion::String; compute_lu::Bool)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:18
 [13] config_from_nc(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:13
 [14] top-level scope
    @ REPL[11]:1

Not sure if the file is corrupted or something!

@b-r-hamilton
Copy link
Collaborator

I looped through all versions

julia> for v in versionlist()
       try 
       config_from_nc(v)
       catch
       println("error with " * v) 
       end
       end

and also found that the LGM_90x45x33_G14A throws an error

julia> config_from_nc("LGM_90x45x33_G14A")
read via GoogleDrive.jl
┌ Info: Downloading
│   source = "https://drive.usercontent.google.com/download?id=1ADkDI3Fc3z4Vm75K5u6hx0Yu1P0iVnW1&export=download"
│   dest = "/home/brynn/Downloads/TMI.jl/data/download"
│   progress = 1.0
│   time_taken = "0.0 s"
│   time_remaining = "0.0 s"
│   average_speed = "1.566 MiB/s"
│   downloaded = "1.604 KiB"
│   remaining = "0 bytes"
└   total = "1.604 KiB"
ERROR: HTTP.Exceptions.StatusError(404, "GET", "/download?id=1ADkDI3Fc3z4Vm75K5u6hx0Yu1P0iVnW1&export=download", HTTP.Messages.Response:
"""
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 04 Sep 2024 16:50:16 GMT
Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factors, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/DriveUntrustedContentHttp/cspreport, script-src 'report-sample' 'nonce-zmd1QuuOzRweP0P5by4AEw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DriveUntrustedContentHttp/cspreport;worker-src 'self'
Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factors=*, ch-ua-platform=*, ch-ua-platform-version=*
Cross-Origin-Opener-Policy: same-origin
Content-Length: 1642
X-GUploader-UploadID: AD-8ljuRuB1NNgmIhT5ps_eE4FIket8cSeheb8dr5WVTya-Z7OF3C171ztUmVAJ59WI9dIshf68
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Security-Policy: sandbox allow-scripts

""")
Stacktrace:
  [1] (::HTTP.ConnectionRequest.var"#connections#4"{…})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{…})
    @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:144
  [2] connections
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:60 [inlined]
  [3] (::Base.var"#96#98"{…})(args::HTTP.Messages.Request; kwargs::@Kwargs{…})
    @ Base ./error.jl:296
  [4] (::HTTP.RetryRequest.var"#manageretries#3"{…})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{…})
    @ HTTP.RetryRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:75
  [5] manageretries
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:30 [inlined]
  [6] (::HTTP.CookieRequest.var"#managecookies#4"{…})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{…})
    @ HTTP.CookieRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/CookieRequest.jl:42
  [7] (::HTTP.HeadersRequest.var"#defaultheaders#2"{…})(req::HTTP.Messages.Request; iofunction::Function, decompress::Bool, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{…})
    @ HTTP.HeadersRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:71
  [8] defaultheaders
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:14 [inlined]
  [9] (::HTTP.RedirectRequest.var"#redirects#3"{…})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{…})
    @ HTTP.RedirectRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:25
 [10] redirects
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:14 [inlined]
 [11] (::HTTP.MessageRequest.var"#makerequest#3"{…})(method::String, url::URIs.URI, headers::Vector{…}, body::Nothing; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{…})
    @ HTTP.MessageRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:35
 [12] makerequest
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:24 [inlined]
 [13] request(stack::HTTP.MessageRequest.var"#makerequest#3"{…}, method::String, url::SubString{…}, h::Vector{…}, b::Nothing, q::Nothing; headers::Vector{…}, body::Nothing, query::Nothing, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:457
 [14] #request#20
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:315 [inlined]
 [15] open
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:579 [inlined]
 [16] download(url::SubString{…}, local_path::String, headers::Vector{…}; update_period::Float32, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/download.jl:106
 [17] download (repeats 2 times)
    @ ~/.julia/packages/HTTP/sJD5V/src/download.jl:96 [inlined]
 [18] #fetch_http#26
    @ ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:80 [inlined]
 [19] fetch_http(remotepath::SubString{String}, localdir::String)
    @ DataDeps ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:78
 [20] google_download(url::String, localdir::String)
    @ GoogleDrive ~/.julia/packages/GoogleDrive/z9zIO/src/GoogleDrive.jl:80
 [21] download_ncfile(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:81
 [22] config_from_nc(TMIversion::String; compute_lu::Bool)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:15
 [23] config_from_nc(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:13
 [24] top-level scope
    @ REPL[13]:1

caused by: HTTP.Exceptions.StatusError(404, "GET", "/download?id=1ADkDI3Fc3z4Vm75K5u6hx0Yu1P0iVnW1&export=download", HTTP.Messages.Response:
"""
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 04 Sep 2024 16:50:16 GMT
Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factors, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/DriveUntrustedContentHttp/cspreport, script-src 'report-sample' 'nonce-zmd1QuuOzRweP0P5by4AEw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DriveUntrustedContentHttp/cspreport;worker-src 'self'
Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factors=*, ch-ua-platform=*, ch-ua-platform-version=*
Cross-Origin-Opener-Policy: same-origin
Content-Length: 1642
X-GUploader-UploadID: AD-8ljuRuB1NNgmIhT5ps_eE4FIket8cSeheb8dr5WVTya-Z7OF3C171ztUmVAJ59WI9dIshf68
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Security-Policy: sandbox allow-scripts

""")
Stacktrace:
  [1] (::HTTP.ExceptionRequest.var"#exceptions#2"{…})(stream::HTTP.Streams.Stream{…}; status_exception::Bool, timedout::Nothing, logerrors::Bool, logtag::Nothing, kw::@Kwargs{…})
    @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ExceptionRequest.jl:19
  [2] exceptions
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ExceptionRequest.jl:13 [inlined]
  [3] (::HTTP.TimeoutRequest.var"#timeouts#3"{…})(stream::HTTP.Streams.Stream{…}; readtimeout::Int64, logerrors::Bool, logtag::Nothing, kw::@Kwargs{…})
    @ HTTP.TimeoutRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/TimeoutRequest.jl:18
  [4] (::HTTP.ConnectionRequest.var"#connections#4"{…})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{…})
    @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:122
  [5] connections
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:60 [inlined]
  [6] (::Base.var"#96#98"{…})(args::HTTP.Messages.Request; kwargs::@Kwargs{…})
    @ Base ./error.jl:296
  [7] (::HTTP.RetryRequest.var"#manageretries#3"{…})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{…})
    @ HTTP.RetryRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:75
  [8] manageretries
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:30 [inlined]
  [9] (::HTTP.CookieRequest.var"#managecookies#4"{…})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{…})
    @ HTTP.CookieRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/CookieRequest.jl:42
 [10] (::HTTP.HeadersRequest.var"#defaultheaders#2"{…})(req::HTTP.Messages.Request; iofunction::Function, decompress::Bool, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{…})
    @ HTTP.HeadersRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:71
 [11] defaultheaders
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:14 [inlined]
 [12] (::HTTP.RedirectRequest.var"#redirects#3"{…})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{…})
    @ HTTP.RedirectRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:25
 [13] redirects
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:14 [inlined]
 [14] (::HTTP.MessageRequest.var"#makerequest#3"{…})(method::String, url::URIs.URI, headers::Vector{…}, body::Nothing; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{…})
    @ HTTP.MessageRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:35
 [15] makerequest
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:24 [inlined]
 [16] request(stack::HTTP.MessageRequest.var"#makerequest#3"{…}, method::String, url::SubString{…}, h::Vector{…}, b::Nothing, q::Nothing; headers::Vector{…}, body::Nothing, query::Nothing, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:457
 [17] #request#20
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:315 [inlined]
 [18] open
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:579 [inlined]
 [19] download(url::SubString{…}, local_path::String, headers::Vector{…}; update_period::Float32, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/download.jl:106
 [20] download (repeats 2 times)
    @ ~/.julia/packages/HTTP/sJD5V/src/download.jl:96 [inlined]
 [21] #fetch_http#26
    @ ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:80 [inlined]
 [22] fetch_http(remotepath::SubString{String}, localdir::String)
    @ DataDeps ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:78
 [23] google_download(url::String, localdir::String)
    @ GoogleDrive ~/.julia/packages/GoogleDrive/z9zIO/src/GoogleDrive.jl:80
 [24] download_ncfile(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:81
 [25] config_from_nc(TMIversion::String; compute_lu::Bool)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:15
 [26] config_from_nc(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:13
 [27] top-level scope
    @ REPL[13]:1
Some type information was truncated. Use `show(err)` to see complete types.

@b-r-hamilton
Copy link
Collaborator

My current workaround for the TMI_modern_180x90x33_GH11_GH12 is to copy the .nc file from a different installed version of TMI to the version being used where I need it.
e.g.
brynn@brynn-ThinkPad-T14-Gen-2i:~/.julia/packages/TMI/NUh7w$ cp data/TMI_modern_180x90x33_GH11_GH12.nc ../ynBgR/data/

@ggebbie
Copy link
Owner Author

ggebbie commented Sep 5, 2024

Thank you for noting these errors. It is the same problem (not properly downloaded due to virus scanner) as noted above, I believe.

@ggebbie
Copy link
Owner Author

ggebbie commented Sep 5, 2024

also see Issue #41

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Downloads.download to get HTTPs nordic link ggebbie/TMI.jl
2 participants
@b-r-hamilton @ggebbie