class Rightscale::HttpConnection
HttpConnection
maintains a persistent HTTP connection to a remote server. Each instance maintains its own unique connection to the HTTP server. HttpConnection
makes a best effort to receive a proper HTTP response from the server, although it does not guarantee that this response contains a HTTP Success code.
On low-level errors (TCP/IP errors) HttpConnection
invokes a reconnect and retry algorithm. Note that although each HttpConnection
object has its own connection to the HTTP server, error handling is shared across all connections to a server. For example, if there are three connections to www.somehttpserver.com, a timeout error on one of those connections will cause all three connections to break and reconnect. A connection will not break and reconnect, however, unless a request becomes active on it within a certain amount of time after the error (as specified by HTTP_CONNECTION_RETRY_DELAY
). An idle connection will not break even if other connections to the same server experience errors.
A HttpConnection
will retry a request a certain number of times (as defined by HTTP_CONNNECTION_RETRY_COUNT). If all the retries fail, an exception is thrown and all HttpConnections associated with a server enter a probationary period defined by HTTP_CONNECTION_RETRY_DELAY
. If the user makes a new request subsequent to entering probation, the request will fail immediately with the same exception thrown on probation entry. This is so that if the HTTP server has gone down, not every subsequent request must wait for a connect timeout before failing. After the probation period expires, the internal state of the HttpConnection
is reset and subsequent requests have the full number of potential reconnects and retries available to them.
Constants
- HTTP_CONNECTION_OPEN_TIMEOUT
Throw a Timeout::Error if a connection isn't established within this number of seconds
- HTTP_CONNECTION_READ_TIMEOUT
Throw a Timeout::Error if no data have been read on this connnection within this number of seconds
- HTTP_CONNECTION_RETRY_COUNT
Number of times to retry the request after encountering the first error
- HTTP_CONNECTION_RETRY_DELAY
Length of the post-error probationary period during which all requests will fail
Attributes
Public Class Methods
# File lib/right_http_connection.rb 422 def self.blank?(obj) 423 case obj 424 when NilClass, FalseClass 425 true 426 when TrueClass, Numeric 427 false 428 when Array, Hash 429 obj.empty? 430 when String 431 obj.empty? || obj.strip.empty? 432 else 433 # "", " ", nil, [], and {} are blank 434 if obj.respond_to?(:empty?) && obj.respond_to?(:strip) 435 obj.empty? or obj.strip.empty? 436 elsif obj.respond_to?(:empty?) 437 obj.empty? 438 else 439 !obj 440 end 441 end 442 end
Params hash:
:user_agent => 'www.HostName.com' # String to report as HTTP User agent :ca_file => 'path_to_file' # A path of a CA certification file in PEM format. The file can contain several CA certificates. :logger => Logger object # If omitted, HttpConnection logs to STDOUT :exception => Exception to raise # The type of exception to raise if a request repeatedly fails. RuntimeError is raised if this parameter is omitted. :http_connection_retry_count # by default == Rightscale::HttpConnection.params[:http_connection_retry_count] :http_connection_open_timeout # by default == Rightscale::HttpConnection.params[:http_connection_open_timeout] :http_connection_read_timeout # by default == Rightscale::HttpConnection.params[:http_connection_read_timeout] :http_connection_retry_delay # by default == Rightscale::HttpConnection.params[:http_connection_retry_delay]
# File lib/right_http_connection.rb 128 def initialize(params={}) 129 @params = params 130 @params[:http_connection_retry_count] ||= @@params[:http_connection_retry_count] 131 @params[:http_connection_open_timeout] ||= @@params[:http_connection_open_timeout] 132 @params[:http_connection_read_timeout] ||= @@params[:http_connection_read_timeout] 133 @params[:http_connection_retry_delay] ||= @@params[:http_connection_retry_delay] 134 @http = nil 135 @server = nil 136 @logger = get_param(:logger) || 137 (RAILS_DEFAULT_LOGGER if defined?(RAILS_DEFAULT_LOGGER)) 138 unless @logger 139 @logger = Logger.new(STDOUT) 140 @logger.level = Logger::INFO 141 end 142 end
Query the global (class-level) parameters:
:user_agent => 'www.HostName.com' # String to report as HTTP User agent :ca_file => 'path_to_file' # Path to a CA certification file in PEM format. The file can contain several CA certificates. If this parameter isn't set, HTTPS certs won't be verified. :logger => Logger object # If omitted, HttpConnection logs to STDOUT :exception => Exception to raise # The type of exception to raise # if a request repeatedly fails. RuntimeError is raised if this parameter is omitted. :http_connection_retry_count # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_COUNT :http_connection_open_timeout # by default == Rightscale::HttpConnection::HTTP_CONNECTION_OPEN_TIMEOUT :http_connection_read_timeout # by default == Rightscale::HttpConnection::HTTP_CONNECTION_READ_TIMEOUT :http_connection_retry_delay # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_DELAY
# File lib/right_http_connection.rb 101 def self.params 102 @@params 103 end
Set the global (class-level) parameters
# File lib/right_http_connection.rb 106 def self.params=(params) 107 @@params = params 108 end
Public Instance Methods
# File lib/right_http_connection.rb 418 def close(reason='') 419 finish 420 end
# File lib/right_http_connection.rb 410 def finish(reason = '') 411 if @http && @http.started? 412 reason = ", reason: '#{reason}'" unless self.class.blank?(reason) 413 @logger.debug("Closing #{@http.use_ssl? ? 'HTTPS' : 'HTTP'} connection to #{@http.address}:#{@http.port}#{reason}") 414 @http.finish 415 end 416 end
# File lib/right_http_connection.rb 144 def get_param(name) 145 @params[name] || @@params[name] 146 end
Set the maximum size (in bytes) of a single read from local data sources like files. This can be used to tune the performance of, for example, a streaming PUT of a large buffer.
# File lib/right_http_connection.rb 172 def local_read_size=(newsize) 173 Net::HTTPGenericRequest.local_read_size=(newsize) 174 end
Query for the maximum size (in bytes) of a single read from local data sources like files. This is important, for example, in a streaming PUT of a large buffer.
# File lib/right_http_connection.rb 165 def local_read_size? 166 Net::HTTPGenericRequest.local_read_size? 167 end
Send HTTP request to server
request_params hash: :server => 'www.HostName.com' # Hostname or IP address of HTTP server :port => '80' # Port of HTTP server :protocol => 'https' # http and https are supported on any port :request => 'requeststring' # Fully-formed HTTP request to make
Raises RuntimeError, Interrupt, and params (if specified in new).
# File lib/right_http_connection.rb 322 def request(request_params, &block) 323 # We save the offset here so that if we need to retry, we can return the file pointer to its initial position 324 mypos = get_fileptr_offset(request_params) 325 loop do 326 # if we are inside a delay between retries: no requests this time! 327 if error_count > @params[:http_connection_retry_count] && 328 error_time + @params[:http_connection_retry_delay] > Time.now 329 # store the message (otherwise it will be lost after error_reset and 330 # we will raise an exception with an empty text) 331 banana_message_text = banana_message 332 @logger.warn("#{err_header} re-raising same error: #{banana_message_text} " + 333 "-- error count: #{error_count}, error age: #{Time.now.to_i - error_time.to_i}") 334 exception = get_param(:exception) || RuntimeError 335 raise exception.new(banana_message_text) 336 end 337 338 # try to connect server(if connection does not exist) and get response data 339 begin 340 request_params[:protocol] ||= (request_params[:port] == 443 ? 'https' : 'http') 341 342 request = request_params[:request] 343 request['User-Agent'] = get_param(:user_agent) || '' 344 345 # (re)open connection to server if none exists or params has changed 346 unless @http && 347 @http.started? && 348 @server == request_params[:server] && 349 @port == request_params[:port] && 350 @protocol == request_params[:protocol] 351 start(request_params) 352 end 353 354 # Detect if the body is a streamable object like a file or socket. If so, stream that 355 # bad boy. 356 setup_streaming(request) 357 response = @http.request(request, &block) 358 359 error_reset 360 eof_reset 361 return response 362 363 # We treat EOF errors and the timeout/network errors differently. Both 364 # are tracked in different statistics blocks. Note below that EOF 365 # errors will sleep for a certain (exponentially increasing) period. 366 # Other errors don't sleep because there is already an inherent delay 367 # in them; connect and read timeouts (for example) have already 368 # 'slept'. It is still not clear which way we should treat errors 369 # like RST and resolution failures. For now, there is no additional 370 # delay for these errors although this may change in the future. 371 372 # EOFError means the server closed the connection on us. 373 rescue EOFError => e 374 @logger.debug("#{err_header} server #{@server} closed connection") 375 @http = nil 376 377 # if we have waited long enough - raise an exception... 378 if raise_on_eof_exception? 379 exception = get_param(:exception) || RuntimeError 380 @logger.warn("#{err_header} raising #{exception} due to permanent EOF being received from #{@server}, error age: #{Time.now.to_i - eof_time.to_i}") 381 raise exception.new("Permanent EOF is being received from #{@server}.") 382 else 383 # ... else just sleep a bit before new retry 384 sleep(add_eof) 385 # We will be retrying the request, so reset the file pointer 386 reset_fileptr_offset(request, mypos) 387 end 388 rescue Exception => e # See comment at bottom for the list of errors seen... 389 @http = nil 390 # if ctrl+c is pressed - we have to reraise exception to terminate proggy 391 if e.is_a?(Interrupt) && !(e.is_a?(Errno::ETIMEDOUT) || e.is_a?(Timeout::Error)) 392 @logger.debug("#{err_header} request to server #{@server} interrupted by ctrl-c") 393 raise 394 elsif e.is_a?(ArgumentError) && e.message.include?('wrong number of arguments (5 for 4)') 395 # seems our net_fix patch was overriden... 396 exception = get_param(:exception) || RuntimeError 397 raise exception.new('incompatible Net::HTTP monkey-patch') 398 end 399 # oops - we got a banana: log it 400 error_add(e.message) 401 @logger.warn("#{err_header} request failure count: #{error_count}, exception: #{e.inspect}") 402 403 # We will be retrying the request, so reset the file pointer 404 reset_fileptr_offset(request, mypos) 405 406 end 407 end 408 end
Set the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.
# File lib/right_http_connection.rb 158 def socket_read_size=(newsize) 159 Net::BufferedIO.socket_read_size=(newsize) 160 end
Query for the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.
# File lib/right_http_connection.rb 151 def socket_read_size? 152 Net::BufferedIO.socket_read_size? 153 end