MIME type detection seems to break on vidly links

[Nolski]

I noticed that when requesting the mimetype for a vidly link (such as https://vid.ly/m9n7s1?content=video&format=webm, node-hubble seems to hang. I think this is due to the fact that the vidly links are shortened and are redirected to a longer link which actually contains the video.

[humphd]

Interesting. This almost seems like a bug in request. Here's a reduced test case that will hang, and is basically what hubble does now:

var request = require('request');
request({
    method: 'HEAD',
    url: "https://vid.ly/m9n7s1?content=video&format=webm",
    followAllRedirects: true
  },
  function(err, res) {
    if (err) {
      console.error(err);
      return;
    }

    console.log({
      href: res.request.href,
      contentType: res.headers['content-type']
    });
  }
);

However, if you remove the followAllRedirects: true line, it will work. It almost seems like it's trying to download the entire video, which is why it takes so long to return. I purposely used the followAllRedirects to beat URL shorteners like this, so I'm not sure what's up here.

I tried updated request to latest, but that's not the issue. I also don't see any bugs related to this in their issues.

[humphd]

Here's what it looks like when run with request.debug=true

node test
REQUEST { method: 'HEAD',
  url: 'https://vid.ly/m9n7s1?content=video&format=webm',
  followAllRedirects: true,
  callback: [Function] }
REQUEST make request https://vid.ly/m9n7s1?content=video&format=webm
REQUEST onRequestResponse https://vid.ly/m9n7s1?content=video&format=webm 302 { 'content-location': 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2',
  'content-type': 'video/webm',
  date: 'Fri, 19 Jun 2015 02:33:59 GMT',
  location: 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2',
  server: 'nginx',
  'x-powered-by': 'PHP/5.3.9',
  connection: 'keep-alive' }
REQUEST redirect https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2
REQUEST redirect to https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2
REQUEST {}
REQUEST response end https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2 302 { 'content-location': 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2',
  'content-type': 'video/webm',
  date: 'Fri, 19 Jun 2015 02:33:59 GMT',
  location: 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2',
  server: 'nginx',
  'x-powered-by': 'PHP/5.3.9',
  connection: 'keep-alive' }
REQUEST make request https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2
REQUEST onRequestResponse https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2 200 { 'content-type': 'video/webm',
  'content-length': '449034465',
  connection: 'keep-alive',
  date: 'Fri, 19 Jun 2015 02:34:01 GMT',
  'last-modified': 'Wed, 17 Jun 2015 23:01:22 GMT',
  etag: '"41d359e5ab6a918756d3c674de550ea1-14"',
  'accept-ranges': 'bytes',
  server: 'AmazonS3',
  'x-cache': 'Miss from cloudfront',
  via: '1.1 0288235ab7012e553755d4e79ab7d1f0.cloudfront.net (CloudFront)',
  'x-amz-cf-id': 'GkrxV7Lk0x_cNTJtoLxpfntr_wP0yXfnc3-h8L3umifiufxTSlwYeA==' }
REQUEST finish init function https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468123955837f97528e2

vs. when it doesn't hang:

node test
REQUEST { method: 'HEAD',
  url: 'https://vid.ly/m9n7s1?content=video&format=webm',
  callback: [Function] }
REQUEST make request https://vid.ly/m9n7s1?content=video&format=webm
REQUEST onRequestResponse https://vid.ly/m9n7s1?content=video&format=webm 302 { 'content-location': 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545',
  'content-type': 'video/webm',
  date: 'Fri, 19 Jun 2015 02:35:43 GMT',
  location: 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545',
  server: 'nginx',
  'x-powered-by': 'PHP/5.3.9',
  connection: 'keep-alive' }
REQUEST redirect https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545
REQUEST redirect to https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545
REQUEST {}
REQUEST response end https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545 302 { 'content-location': 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545',
  'content-type': 'video/webm',
  date: 'Fri, 19 Jun 2015 02:35:43 GMT',
  location: 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545',
  server: 'nginx',
  'x-powered-by': 'PHP/5.3.9',
  connection: 'keep-alive' }
REQUEST make request https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545
REQUEST onRequestResponse https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545 200 { 'content-type': 'video/webm',
  'content-length': '449034465',
  connection: 'keep-alive',
  date: 'Fri, 19 Jun 2015 02:15:08 GMT',
  'last-modified': 'Wed, 17 Jun 2015 23:01:22 GMT',
  etag: '"41d359e5ab6a918756d3c674de550ea1-14"',
  'accept-ranges': 'bytes',
  server: 'AmazonS3',
  age: '1236',
  'x-cache': 'Hit from cloudfront',
  via: '1.1 391dfa5542d8d96966e1c02fe2c44aa9.cloudfront.net (CloudFront)',
  'x-amz-cf-id': 'YIrjOvvW7gzBL1Duq7WWA_R6d317QQctYeZUf7VLT1ptw_RLcxfqfA==' }
REQUEST finish init function https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545
REQUEST response end https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545 200 { 'content-type': 'video/webm',
  'content-length': '449034465',
  connection: 'keep-alive',
  date: 'Fri, 19 Jun 2015 02:15:08 GMT',
  'last-modified': 'Wed, 17 Jun 2015 23:01:22 GMT',
  etag: '"41d359e5ab6a918756d3c674de550ea1-14"',
  'accept-ranges': 'bytes',
  server: 'AmazonS3',
  age: '1236',
  'x-cache': 'Hit from cloudfront',
  via: '1.1 391dfa5542d8d96966e1c02fe2c44aa9.cloudfront.net (CloudFront)',
  'x-amz-cf-id': 'YIrjOvvW7gzBL1Duq7WWA_R6d317QQctYeZUf7VLT1ptw_RLcxfqfA==' }
REQUEST end event https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545
REQUEST emitting complete https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545
{ href: 'https://d3fenhwk93s16g.cloudfront.net/m9n7s1/webm.webm?t=143468134255837ffe47545',
  contentType: 'video/webm' }

[humphd]

Filed upstream: request/request#1649

[Nolski]

Looks like this fix should be merged into master now. request/request#1651

[humphd]

They haven't published this yet, still at 2.58.1. Once they republish, I'll update.